by erickrf

erickrf / autoencoder

Text autoencoder with LSTMs

216 Stars 80 Forks Last release: Not found MIT License 10 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Text Autoencoder

This is an implementation of a recurrent neural network that reads an input text, encodes it in its memory cell, and then reconstructs the inputs. This is basically the idea presented by

Sutskever et al. (2014) 

Why? The point of training an autoencoder is to make an RNN learn how to compress a relatively long sequence into a limited, dense vector. Once we have a fixed-size representation of a sentence, there's a lot we can do with it.

We can work with single sentences (classifying them with respect to sentiment, topic, authorship, etc), or more than one at a time (checking for similarities, contradiction, question/answer pairs, etc.) Another successful application is to encode one sentence in one language and use a different autoencoder to decode it into another language, e.g.

Cho et al. (2014) 


The autoencoder is implemented with

_. Specifically, it uses a bidirectional LSTM (but it can be configured to use a simple LSTM instead).

In the encoder step, the LSTM reads the whole input sequence; its outputs at each time step are ignored.

Then, in the decoder step, a special symbol GO is read, and the output of the LSTM is fed to a linear layer with the size of the vocabulary. The chosen word (i.e., the one with the highest score) is the next input to the decoder. This goes on until a special symbol EOS is produced.

The weights of the encoder and decoder are shared.

Performance notes ^^^^^^^^^^^^^^^^^

  • Even for small vocabularies (a few thousand words), training the network over all possible outputs at each time step is very expensive computationally. Instead, we just sample the weights of 100 possible words. During inference time, there is no way around it, but the computational cost is much lesser.

  • For better decoder performance, a beam search is preferable to the currently used greedy choice.


    : reads a text file and create numpy files that can be used to train an autoencoder
    : train a new autoencoder model
    : run a trained autoencoder that reads input from stdin. It can be fun to test the boundaries of your trained model :)
    : run the encoder part of a trained autoencoder on sentences read from a text file. The encoded representation is saved as a numpy file

You can run any of the scripts with

to get information about what arguments they accept.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.