Need help with AssociativeRetrieval?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

jxwufan
127 Stars 15 Forks Apache License 2.0 39 Commits 2 Opened issues

Description

TensorFlow implementation of Fast Weights

Services available

!
?

Need anything else?

Contributors list

Using Fast Weights to Attend to the Recent Past

This repo is a TensorFlow implementation of

Using Fast Weights to Attend to the Recent Past
Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu
NIPS 2016, https://arxiv.org/abs/1610.06258

Specifically, we follow the experiments in

Sec 4.1 Associative retrieval
and try to reproduce the results in Table 1 and Figure 2. The fast weights model can achieve 100% accuracy (0% error rate) on R=50 setting in ~30K iterations.

Running result as follows:

Fast Weights(with layernorm):

Fast Weights(without layernorm):

LSTM(with layernorm):

LSTM(without layernorm):

Both trained on GTX 980 Ti, with TensorFlow 0.11rc1.

Setting on R=50, using ADAM optimizer with default parameters.

Train the fast weights model

python FW_train.py

Evaluate the fast weights model

python FW_eval.py

Run the

LSTM
baseline model in similar ways.

Author

Fan Wu ([email protected])

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.