Off-Line Math Formula Recognition Using Deep Neural Networks

Based on Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition.

Requirements

• Python 3
• PyTorch

All dependencies can be installed with PIP.

pip install -r requirements.txt

If you'd like to use a different installation method or another CUDA version with PyTorch (e.g. CUDA 10) follow the instructions on PyTorch - Getting Started.

Data

CROHME: Competition on Recognition of Online Handwritten Mathematical Expressions has been used. As it is an on-line handwritten dataset, it consists of InkML files, but this architecture is for off-line recognition, which means that images are used as input.

The dataset has been converted to images of size

256x256

The data needs to be in the

data/

tokens.tsv

gt_split/train.tsv

gt_split/validation.tsv

The training/validation split can be generated by running:

python data_tools/train_validation_split.py -i data/groundtruth_train.tsv -o data/gt_split

Note: The content of the generated images vary greatly in size. As longer expressions are limited to the same width, they will essentially use a smaller font. This makes it much more difficult to correctly predict the sequences, especially since the dataset is quite small. The primary focus was the attention mechanism, to see whether it can handle different sizes. If you want better results, the images need to be normalised.

Usage

Training

Training is done with the

train.py

python train.py --prefix "some-name-" -n 200 -c checkpoints/example-0022.pth

The

--prefix

-c

For all options see

python train.py --help

Evaluation

To evaluate a model use the

evaluate.py

For example to evaluate the sets 2014 and 2016 with beam width 5:

python evaluate.py -d 2014 2016 --beam-width 5 -c checkpoints/example-0022.pth