TensorFlow implementation of Fast Weights
This repo is a TensorFlow implementation of
Using Fast Weights to Attend to the Recent Past Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu NIPS 2016, https://arxiv.org/abs/1610.06258
Specifically, we follow the experiments in
Sec 4.1 Associative retrievaland try to reproduce the results in Table 1 and Figure 2. The fast weights model can achieve 100% accuracy (0% error rate) on R=50 setting in ~30K iterations.
Running result as follows:
Fast Weights(with layernorm):
Fast Weights(without layernorm):
Both trained on GTX 980 Ti, with TensorFlow 0.11rc1.
Setting on R=50, using ADAM optimizer with default parameters.
LSTMbaseline model in similar ways.
Fan Wu ([email protected])