Introduction

Note: there is now a PyTorch version of this toolkit (fairseq-py) and new development efforts will focus on it. The Lua version is preserved here, but is provided without any support.

This is fairseq, a sequence-to-sequence learning toolkit for Torch from Facebook AI Research tailored to Neural Machine Translation (NMT). It implements the convolutional NMT models proposed in Convolutional Sequence to Sequence Learning and A Convolutional Encoder Model for Neural Machine Translation as well as a standard LSTM-based model. It features multi-GPU training on a single machine as well as fast beam search generation on both CPU and GPU. We provide pre-trained models for English to French, English to German and English to Romanian translation.

Citation

If you use the code in your paper, then please cite it as:

@article{gehring2017convs2s,
  author          = {Gehring, Jonas and Auli, Michael and Grangier, David and Yarats, Denis and Dauphin, Yann N},
  title           = "{Convolutional Sequence to Sequence Learning}",
  journal         = {ArXiv e-prints},
  archivePrefix   = "arXiv",
  eprinttype      = {arxiv},
  eprint          = {1705.03122},
  primaryClass    = "cs.CL",
  keywords        = {Computer Science - Computation and Language},
  year            = 2017,
  month           = May,
}

and

@article{gehring2016convenc,
  author          = {Gehring, Jonas and Auli, Michael and Grangier, David and Dauphin, Yann N},
  title           = "{A Convolutional Encoder Model for Neural Machine Translation}",
  journal         = {ArXiv e-prints},
  archivePrefix   = "arXiv",
  eprinttype      = {arxiv},
  eprint          = {1611.02344},
  primaryClass    = "cs.CL",
  keywords        = {Computer Science - Computation and Language},
  year            = 2016,
  month           = Nov,
}

Requirements and Installation

• A computer running macOS or Linux
• For training new models, you'll also need a NVIDIA GPU and NCCL
• A Torch installation. For maximum speed, we recommend using LuaJIT and Intel MKL.
• A recent version nn. The minimum required version is from May 5th, 2017. A simple

  luarocks install nn

  is sufficient to update your locally installed version.

Install fairseq by cloning the GitHub repository and running

luarocks make rocks/fairseq-scm-1.rockspec

luarocks make rocks/fairseq-cpu-scm-1.rockspec

The LuaRocks installation provides a command-line tool that includes the following functionality: *

fairseq preprocess

fairseq train

fairseq generate

fairseq generate-lines

fairseq score

fairseq tofloat

fairseq optimize-fconv

-fconvfast

Quick Start

Training a New Model

Data Pre-processing

The fairseq source distribution contains an example pre-processing script for the IWSLT14 German-English corpus. Pre-process and binarize the data as follows:

$ cd data/
$ bash prepare-iwslt14.sh
$ cd ..
$ TEXT=data/iwslt14.tokenized.de-en
$ fairseq preprocess -sourcelang de -targetlang en \
  -trainpref $TEXT/train -validpref $TEXT/valid -testpref $TEXT/test \
  -thresholdsrc 3 -thresholdtgt 3 -destdir data-bin/iwslt14.tokenized.de-en

Training

Use

fairseq train

Standard bi-directional LSTM model

$ mkdir -p trainings/blstm $ fairseq train -sourcelang de -targetlang en -datadir data-bin/iwslt14.tokenized.de-en \ -model blstm -nhid 512 -dropout 0.2 -dropout_hid 0 -optim adam -lr 0.0003125 -savedir trainings/blstm

Fully convolutional sequence-to-sequence model

$ mkdir -p trainings/fconv $ fairseq train -sourcelang de -targetlang en -datadir data-bin/iwslt14.tokenized.de-en \ -model fconv -nenclayer 4 -nlayer 3 -dropout 0.2 -optim nag -lr 0.25 -clip 0.1 \ -momentum 0.99 -timeavg -bptt 0 -savedir trainings/fconv

Convolutional encoder, LSTM decoder

$ mkdir -p trainings/convenc $ fairseq train -sourcelang de -targetlang en -datadir data-bin/iwslt14.tokenized.de-en \ -model conv -nenclayer 6 -dropout 0.2 -dropout_hid 0 -savedir trainings/convenc ```

By default,

fairseq train

-ngpus

Generation

Once your model is trained, you can translate with it using

fairseq generate

fairseq generate-lines

Optional: optimize for generation speed

$ fairseq optimize-fconv -inputmodel trainings/fconv/modelbest.th7 -outputmodel trainings/fconv/modelbest_opt.th7

Translate some text

$ DATA=data-bin/iwslt14.tokenized.de-en $ fairseq generate-lines -sourcedict $DATA/dict.de.th7 -targetdict $DATA/dict.en.th7 \ -path trainings/fconv/modelbestopt.th7 -beam 10 -nbest 2 | [target] Dictionary: 24738 types | [source] Dictionary: 35474 types

eine sprache ist ausdruck des menschlichen geistes . S eine sprache ist ausdruck des menschlichen geistes . O eine sprache ist ausdruck des menschlichen geistes . H -0.23804219067097 a language is expression of human mind . A 2 2 3 4 5 6 7 8 9 H -0.23861141502857 a language is expression of the human mind . A 2 2 3 4 5 7 6 7 9 9 ```

CPU Generation

Use

fairseq tofloat

Optional: optimize for generation speed

$ fairseq optimize-fconv -inputmodel trainings/fconv/modelbest.th7 -outputmodel trainings/fconv/modelbest_opt.th7

Convert to float

$ fairseq tofloat -inputmodel trainings/fconv/modelbestopt.th7 \ -outputmodel trainings/fconv/modelbestopt-float.th7

Translate some text

$ fairseq generate-lines -sourcedict $DATA/dict.de.th7 -targetdict $DATA/dict.en.th7 \ -path trainings/fconv/modelbestopt-float.th7 -beam 10 -nbest 2

eine sprache ist ausdruck des menschlichen geistes . S eine sprache ist ausdruck des menschlichen geistes . O eine sprache ist ausdruck des menschlichen geistes . H -0.2380430996418 a language is expression of human mind . A 2 2 3 4 5 6 7 8 9 H -0.23861189186573 a language is expression of the human mind . A 2 2 3 4 5 7 6 7 9 9 ```

Pre-trained Models

Generation with the binarized test sets can be run in batch mode as follows, e.g. for English-French on a GTX-1080ti: ``` $ fairseq generate -sourcelang en -targetlang fr -datadir data-bin/wmt14.en-fr -dataset newstest2014 \ -path wmt14.en-fr.fconv-cuda/model.th7 -beam 5 -batchsize 128 | tee /tmp/gen.out ... | Translated 3003 sentences (95451 tokens) in 136.3s (700.49 tokens/s) | Timings: setup 0.1s (0.1%), encoder 1.9s (1.4%), decoder 108.9s (79.9%), searchresults 0.0s (0.0%), searchprune 12.5s (9.2%) | BLEU4 = 43.43, 68.2/49.2/37.4/28.8 (BP=0.996, ratio=1.004, syslen=92087, reflen=92448)

Word-level BLEU scoring:

$ grep ^H /tmp/gen.out | cut -f3- | sed 's/@@ //g' > /tmp/gen.out.sys $ grep ^T /tmp/gen.out | cut -f2- | sed 's/@@ //g' > /tmp/gen.out.ref $ fairseq score -sys /tmp/gen.out.sys -ref /tmp/gen.out.ref BLEU4 = 40.55, 67.6/46.5/34.0/25.3 (BP=1.000, ratio=0.998, syslen=81369, reflen=81194) ```

Join the fairseq community

• Facebook page: https://www.facebook.com/groups/fairseq.users
• Google group: https://groups.google.com/forum/#!forum/fairseq-users
• Contact: [email protected], [email protected]

License

fairseq is BSD-licensed. The license applies to the pre-trained models as well. We also provide an additional patent grant.