AssociativeRetrieval

by jxwufan

TensorFlow implementation of Fast Weights

127 Stars 15 Forks Last release: Not found Apache License 2.0 39 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Using Fast Weights to Attend to the Recent Past

This repo is a TensorFlow implementation of

Using Fast Weights to Attend to the Recent Past
Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu
NIPS 2016, https://arxiv.org/abs/1610.06258

Specifically, we follow the experiments in

Sec 4.1 Associative retrieval
and try to reproduce the results in Table 1 and Figure 2. The fast weights model can achieve 100% accuracy (0% error rate) on R=50 setting in ~30K iterations.

Running result as follows:

Fast Weights(with layernorm):

Fast Weights(without layernorm):

LSTM(with layernorm):

LSTM(without layernorm):

Both trained on GTX 980 Ti, with TensorFlow 0.11rc1.

Setting on R=50, using ADAM optimizer with default parameters.

Train the fast weights model

python FW_train.py

Evaluate the fast weights model

python FW_eval.py

Run the

LSTM
baseline model in similar ways.

Author

Fan Wu ([email protected])

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.