Keras Interface for Kaldi ASR
This code interfaces Kaldi tools for Speech Recognition and Keras tools for Deep Learning. Keras simplifies the latest deep learning implementations, unifies the two popular Theano and Tensorflow libraries, and has a growing user base. Kaldi, one of the best tools for ASR, thus needs an interface with Keras tools, and here is one. This code directly interacts with Kaldi style directories of data and alignments to build and test Deep Learning models in Keras.
Trains DNNs from Kaldi GMM system
Works with standard Kaldi data and alignment directories
Supports mini-batch training
Supports LSTMs, maxout and dropout training
Easily extendable to other deep learning implementations in Keras
Decodes test utterances in Kaldi style
Keras with Tensorflow/Theano backend
Train a GMM system in Kaldi. Place stepskt and runkt.sh in the working directory. Configure and run runkt.sh. To train LSTMs, run runkt_LSTM.sh.
train.py is the Keras training script. DNN structure (type of network, activations, number of hidden layers and nodes) can be configured in this script. train_LSTM.py trains LSTMs.
dataGenerator.py provides an object that reads Kaldi data and alignment directories in batches and retrieves mini-batches for training. dataGenSequences.py retrieves 3D mini-batches for LSTM training.
nnet-forward.py passes test features through the trained DNNs and outputs log probabilities (log of DNN outputs) in Kaldi format. nnet-forward-seq.py passes 3D arrays to LSTMs and outputs log probabilities.
kaldiIO.py reads and writes Kaldi-type binary features.
decode.py is the decoding script. decode_seq.py is the script for LSTMs.
align.sh is the alignment script.
compute_priors.py computes priors.
saveModelNnet3.sh and saveModelNnet3Raw.py convert the trained feedforward DNNs into Kaldi's nnet3 format. They currently have limited functionality.
The script uses stochastic gradient descent with 0.5 momentum. It starts with a learning rate of 0.1 for a minimum of 5 iterations. When the validation loss reduces by less than 0.002 between successive iterations, learning rate is halved, and is contined to be halved after each epoch, 18 times.
Timit database of 8 kHz sampling rate was used to train monophone, triphone (300 pdfs), LDA+MLLT (500 pdfs), DNN and LSTM models. Phone error rates are as follows:
DNN (3 hidden layers of 1024 nodes, ReLU activations): 23.71%
LSTM (3 hidden layers of 256 nodes, Tanh activations, LDA+MLLT alignments): 23.02%
WSJ SI-284 corpus of 8 kHz sampling rate was used to train monophone, triphone (1000 pdfs), DNN and LSTM models. Word error rates are as follows:
Monophone - dev93: 37.76%, eval92: 27.95%
Triphone - dev93: 23.78%, eval92: 16.37%
DNN (3 hidden layers of 1024 nodes, ReLU activations) - dev93: 13.50%, eval92: 9.16%
LSTM (3 hidden layers of 256 nodes, ReLU activations) - dev93: 13.25%, eval92: 9.16%
If using ReLU activations in LSTM, use Tensorflow backend.
Initialise Tensorflow with the correct GPU memory fraction.
D S Pavan Kumar
dspavankumar [at] gmail [dot] com
Thanks to Dan Povey, Ram Sundaram, Naresh Kumar, Tejas Godambe and Veera Raghavendra for suggesting improvements and debugging.
GNU GPL v3