Neural Network Models for Joint POS Tagging and Dependency Parsing

Implementations of joint models for POS tagging and dependency parsing, as described in my papers:

1. Dat Quoc Nguyen and Karin Verspoor. 2018. An improved neural network model for joint POS tagging and dependency parsing. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 81-91. [.bib] (jPTDP v2.0)
2. Dat Quoc Nguyen, Mark Dras and Mark Johnson. 2017. A Novel Neural Network Model for Joint POS Tagging and Graph-based Dependency Parsing. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 134-142. [.bib] (jPTDP v1.0)

This github project currently supports jPTDP v2.0, while v1.0 can be found in the release section. Please cite paper [1] when jPTDP is used to produce published results or incorporated into other software. I would highly appreciate to have your bug reports, comments and suggestions about jPTDP. As a free open-source implementation, jPTDP is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

Installation

jPTDP requires the following software packages:

• Python 2.7

• DyNet

  v2.0

  $ virtualenv -p python2.7 .DyNet $ source .DyNet/bin/activate $ pip install cython numpy $ pip install dynet==2.0.3

Once you installed the prerequisite packages above, you can clone or download (and then unzip) jPTDP. Next sections show instructions to train a new joint model for POS tagging and dependency parsing, and then to utilize a pre-trained model.

NOTE: jPTDP is also ported to run with Python 3.4+ by Santiago Castro. Also note that pre-trained models I provide in the last section would not work with this ported version (see a discussion). Thus, you may want to retrain jPTDP if using this ported version.

Train a joint model

Suppose that

SOURCE_DIR

train.conllu

dev.conllu

SOURCE_DIR/sample

To train a joint model for POS tagging and dependency parsing, you perform:

SOURCE_DIR$ python jPTDP.py --dynet-seed 123456789 [--dynet-mem ] [--epochs ] [--lstmdims ] [--lstmlayers ] [--hidden ] [--wembedding ] [--cembedding ] [--pembedding ] [--prevectors ] [--model ] [--params ] --outdir  --train   --dev 

where hyper-parameters in [] are optional:

• --dynet-mem

  : Specify DyNet memory in MB.

• --epochs

  : Specify number of training epochs. Default value is 30.

• --lstmdims

  : Specify number of BiLSTM dimensions. Default value is 128.

• --lstmlayers

  : Specify number of BiLSTM layers. Default value is 2.

• --hidden

  : Specify size of MLP hidden layer. Default value is 100.

• --wembedding

  : Specify size of word embeddings. Default value is 100.

• --cembedding

  : Specify size of character embeddings. Default value is 50.

• --pembedding

  : Specify size of POS tag embeddings. Default value is 100.

• --prevectors

  : Specify path to the pre-trained word embedding file for initialization. Default value is "None" (i.e. word embeddings are randomly initialized).

• --model

  : Specify a name for model parameters file. Default value is "model".

• --params

  : Specify a name for model hyper-parameters file. Default value is "model.params".

• --outdir

  : Specify path to directory where the trained model will be saved.

• --train

  : Specify path to the training data file.

• --dev

  : Specify path to the development data file.

For example:

SOURCE_DIR$ python jPTDP.py --dynet-seed 123456789 --dynet-mem 1000 --epochs 30 --lstmdims 128 --lstmlayers 2 --hidden 100 --wembedding 100 --cembedding 50 --pembedding 100  --model trialmodel --params trialmodel.params --outdir sample/ --train sample/train.conllu --dev sample/dev.conllu

will produce model files

trialmodel

trialmodel.params

SOURCE_DIR/sample

If you would like to use the fine-grained language-specific POS tags in the 5th column instead of the coarse-grained POS tags in the 4th column, you should use

swapper.py

SOURCE_DIR/utils

SOURCE_DIR$ python utils/swapper.py 

For example:

SOURCE_DIR$ python utils/swapper.py sample/train.conllu
SOURCE_DIR$ python utils/swapper.py sample/dev.conllu

will generate two new files for training:

train.conllu.ux2xu

dev.conllu.ux2xu

SOURCE_DIR/sample

Utilize a pre-trained model

Assume that you are going to utilize a pre-trained model for annotating a corpus whose each line represents a tokenized/word-segmented sentence. You should use

converter.py

SOURCE_DIR/utils

SOURCE_DIR$ python utils/converter.py 

For example:

SOURCE_DIR$ python utils/converter.py sample/test

will generate in folder

SOURCE_DIR/sample

test.conllu

To utilize a pre-trained model for POS tagging and dependency parsing, you perform:

SOURCE_DIR$ python jPTDP.py --predict --model  --params  --test  --outdir  --output 

• --model

  : Specify path to model parameters file.

• --params

  : Specify path to model hyper-parameters file.

• --test

  : Specify path to 10-column input file.

• --outdir

  : Specify path to directory where output file will be saved.

• --output

  : Specify name of the output file.

For example:

SOURCE_DIR$ python jPTDP.py --predict --model sample/trialmodel --params sample/trialmodel.params --test sample/test.conllu --outdir sample/ --output test.conllu.pred
SOURCE_DIR$ python jPTDP.py --predict --model sample/trialmodel --params sample/trialmodel.params --test sample/dev.conllu --outdir sample/ --output dev.conllu.pred

will produce output files

test.conllu.pred

dev.conllu.pred

SOURCE_DIR/sample

Pre-trained models

Pre-trained jPTDP v2.0 models, which were trained on English WSJ Penn treebank, GENIA and UD v2.2 treebanks, can be found at HERE. Results on test sets (as detailed in paper [1]) are as follows:

model256

model

model256