Need help with autoencoder?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

erickrf
233 Stars 88 Forks MIT License 10 Commits 2 Opened issues

Description

Text autoencoder with LSTMs

Services available

!
?

Need anything else?

Contributors list

# 100,942
Python
Shell
Natural...
pos-tag...
5 commits
# 466,571
Python
1 commit
# 101,678
Shell
Jupyter...
Clojure
gRPC
1 commit

Text Autoencoder

This is an implementation of a recurrent neural network that reads an input text, encodes it in its memory cell, and then reconstructs the inputs. This is basically the idea presented by

Sutskever et al. (2014) 
_

Why? The point of training an autoencoder is to make an RNN learn how to compress a relatively long sequence into a limited, dense vector. Once we have a fixed-size representation of a sentence, there's a lot we can do with it.

We can work with single sentences (classifying them with respect to sentiment, topic, authorship, etc), or more than one at a time (checking for similarities, contradiction, question/answer pairs, etc.) Another successful application is to encode one sentence in one language and use a different autoencoder to decode it into another language, e.g.

Cho et al. (2014) 
_.

Implementation

The autoencoder is implemented with

Tensorflow 
_. Specifically, it uses a bidirectional LSTM (but it can be configured to use a simple LSTM instead).

In the encoder step, the LSTM reads the whole input sequence; its outputs at each time step are ignored.

Then, in the decoder step, a special symbol GO is read, and the output of the LSTM is fed to a linear layer with the size of the vocabulary. The chosen word (i.e., the one with the highest score) is the next input to the decoder. This goes on until a special symbol EOS is produced.

The weights of the encoder and decoder are shared.

Performance notes ^^^^^^^^^^^^^^^^^

  • Even for small vocabularies (a few thousand words), training the network over all possible outputs at each time step is very expensive computationally. Instead, we just sample the weights of 100 possible words. During inference time, there is no way around it, but the computational cost is much lesser.

  • For better decoder performance, a beam search is preferable to the currently used greedy choice.

Scripts

  • prepare-data.py
    : reads a text file and create numpy files that can be used to train an autoencoder
  • train-autoencoder.py
    : train a new autoencoder model
  • interactive.py
    : run a trained autoencoder that reads input from stdin. It can be fun to test the boundaries of your trained model :)
  • codify-sentences.py
    : run the encoder part of a trained autoencoder on sentences read from a text file. The encoded representation is saved as a numpy file

You can run any of the scripts with

-h
to get information about what arguments they accept.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.