by jxwufan

TensorFlow implementation of Fast Weights

127 Stars 15 Forks Last release: Not found Apache License 2.0 39 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Using Fast Weights to Attend to the Recent Past

This repo is a TensorFlow implementation of

Using Fast Weights to Attend to the Recent Past
Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu
NIPS 2016,

Specifically, we follow the experiments in

Sec 4.1 Associative retrieval
and try to reproduce the results in Table 1 and Figure 2. The fast weights model can achieve 100% accuracy (0% error rate) on R=50 setting in ~30K iterations.

Running result as follows:

Fast Weights(with layernorm):

Fast Weights(without layernorm):

LSTM(with layernorm):

LSTM(without layernorm):

Both trained on GTX 980 Ti, with TensorFlow 0.11rc1.

Setting on R=50, using ADAM optimizer with default parameters.

Train the fast weights model


Evaluate the fast weights model


Run the

baseline model in similar ways.


Fan Wu ([email protected])

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.