Automatic Speech Recognition (ASR) - DeepSpeech German

This is the project for the paper German End-to-end Speech Recognition based on DeepSpeech published at KONVENS 2019.

This project aims to develop a working Speech to Text module using Mozilla DeepSpeech, which can be used for any Audio processing pipeline. Mozilla DeepSpeech Architecture is a state-of-the-art open-source automatic speech recognition (ASR) toolkit. DeepSpeech is using a model trained by machine learning techniques based on Baidu's Deep Speech research paper. Project DeepSpeech uses Google's TensorFlow to make the implementation easier.

Important Links:

Paper: https://www.researchgate.net/publication/336532830GermanEnd-to-endSpeechRecognitionbasedon_DeepSpeech

DeepSpeech-API: https://github.com/AASHISHAG/DeepSpeech-API

This Readme is written for DeepSpeech v0.9.3. Refer to Mozillla DeepSpeech for latest updates.

Contents

1. Requirements
2. Speech Corpus
3. Language Model
4. Training
5. Hyper-Parameter Optimization
6. Results
7. Trained Models
8. Acknowledgments
9. References

Requirements

The instruction are focusing on Linux. It might be possible to use also MacOS and Windows as long as a shell environment can be provided, however some instructions and scripts might have to be adjusted then.

DeepSpeech

KenLM

git pull --recurse-submodules

Developer Information

To update the used DeepSpeech version checkout in the

DeepSpeech

git checkout tags/v0.9.3

The same applies to KenLM.

Installing Python bindings

The DeepSpeech tools require still TensorFlow 1.15, which is only supported up to Python 3.7.

pyenv

git clone https://github.com/pyenv/pyenv.git $HOME/.pyenv

Follow the instruction on the pyenv page, for Ubuntu Desktop Add to your

~/.bashrc

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc

echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bash_profile

Log out or run:

source ~/.bashrc

virtualenv

pyenv install 3.7.9
sudo pip3 install virtualenv
virtualenv --python=$HOME/.pyenv/versions/3.7.9/bin/python3.7 python-environments
source python-environments/bin/activate
pip3 install -r python_requirements.txt

NOTE: While writing this 2 issues were fixed in audiomate and a GitHub patched version was used. This can be updated once the upstream audiomate has released a new version.

Installing Linux dependencies

The necessary Linux dependencies can be found in

linux_requirements

xargs -a linux_requirements.txt sudo apt-get install

Speech Corpus

• German Distant Speech Corpus (TUDA-De) ~127h
  • Download Folder
• Mozilla Common Voice ~140h
  • Download
• Voxforge ~35h
  • Download Folder
• M-AILABS
  • Download Folder
• Spoken Wikipedia Corpora
  • Download Folder

Download the corpus

For the speech corpus download the script

download_speech_corpus.py

source python-environments/bin/activate
python pre-processing/download_speech_corpus.py --tuda --cv --swc --voxforge --mailabs

NOTE: The python module of

audiomate

audiomate

is patching these URLS, but in case a new speech corpus is released 
the URLs in the script must be updated again or passed as arguments

, e.g.

It is also possible to leave out some corpus and specify just a subset to download. The default target directory is the current directory, i.e. project root but could be set with

--target_path

The tool will display after a while a download progress indicator.

The downloads will be found in the folder

tuda

mozilla

voxforge

swc

mailabs

Prepare the Audio Data

The output of this step is a CSV format with references to the audio and corresponding text split into train, dev and test data for the TensorFlow network.

Convert to UTF-8:

pre-processing/run_to_utf_8.sh

The

pre-processing/prepare_data

Execute the following commands in the root directory of this project after running:

source python-environments/bin/activate

NOTE: If the filename or folders of the downloaded speech corpora are changing the paths used in this section must be updated, too.

Some Examples:

1. Tuda-De

python pre-processing/prepare_data.py --tuda tuda/german-speechdata-package-v3 german-speech-corpus/data_tuda

2. Mozilla

python pre-processing/prepare_data.py --cv mozilla/cv-corpus-6.1-2020-12-11/de german-speech-corpus/data_mozilla

3. Voxforge

python pre-processing/prepare_data.py --voxforge voxforge german-speech-corpus/data_voxforge

4. Voxforge + Tuda-De

python pre-processing/prepare_data.py --voxforge voxforge --tuda tuda/german-speechdata-package-v2 german-speech-corpus/data_tuda+voxforge

5. Voxforge + Tuda-De + Mozilla

python pre-processing/prepare_data.py --cv mozilla/cv-corpus-6.1-2020-12-11/de/clips --voxforge voxforge --tuda tuda/german-speechdata-package-v3 german-speech-corpus/data_tuda+voxforge+mozilla

Language Model

We used KenLM toolkit to train a 3-gram language model. It is Language Model inference code by Kenneth Heafield Consult the repository for any information regarding the compilation in case of errors.

• Installation

cd kenlm
mkdir -p build
cd build
cmake ..
make -j `nproc`

nproc

Text Corpus

We used an open-source German Speech Corpus released by University of Hamburg.

Download the data:

mkdir german-text-corpus
cd german-text-corpus
wget http://ltdata1.informatik.uni-hamburg.de/kaldi_tuda_de/German_sentences_8mil_filtered_maryfied.txt.gz
gzip -d German_sentences_8mil_filtered_maryfied.txt.gz

Pre-process the data

Execute the following commands in the root directory of this project.

source python-environments/bin/activate
python pre-processing/prepare_vocab.py german-text-corpus/German_sentences_8mil_filtered_maryfied.txt german-text-corpus/clean_vocab.txt

Build the Language Model

This step will create a

trie

create_language_model.sh

The script will use the compiled KenLM binaries and download a native DeepSpeechClient. The native DeepSpeech client is using the "cpu" architecture. For a different architecture pass as additional parameter the architecture. See the

--arch

python DeepSpeech/util/taskcluster.py --help

The mandatory

top_k

The

default_alpha

default_beta

DeepSpeech.py

lm_optimizer.py

./create_language_model.sh

NOTE: A different scorer might be useful later for the efficient speech recognition in case of a special language domain. Then an own text corpus should be used, and the script source can be used as a template for this scorer and be executed separately.

Training

NOTE: The training can be accelerated by using CUDA. Follow the GPU Installation Instructions for TensorFlow. At the time of writing also on Ubuntu 20.04 the toolchain for Ubuntu 18.04 has to be used.

If CUDA is used the GPU package of TensorFlow has to be installed:

source python-environments/bin/activate
pip uninstall tensorflow
pip install tensorflow-gpu==1.15.4

nohup ./train_model.sh  &

Example:

nohup ./train_model.sh german-speech-corpus/data_tuda+voxforge+mozilla &

Parameters:

• The default scorer is

  german-text-corpus/kenlm.scorer

  .
• The default alphabet is

  data/alphabet.txt

  .
• The default DeepSpeech processing data is stored under

  deepspeech_processing

  .
• The model is stored under the directory

  models

  .

It is possible to modify these parameters. If a parameter should take the default value, pass

""

./train_model.sh     

Example:

./train_model.sh german-speech-corpus/data_tuda+voxforge+mozilla "" "" my_processing_dir my_models

NOTE: The augmentation parameters are not used and might be interesting for improving the results.

NOTE: Increase the `exportmodelversion

option in

NOTE: In case a

Not enough time for target transition sequence (required: 171, available: 0)

and
add

to the call to

TFLite Model Generation

The TensorFLow Lite model suitable for resource restricted devices (embedded, Raspberry Pi 4, ...) can be generated with:

./export_tflite.sh

Parameters:

• The default alphabet is

  data/alphabet.txt

  .
• The default DeepSpeech processing data is taken from

  deepspeech_processing

  .
• The model is stored under the directory

  models

  .

It is possible to modify these parameters. If a parameter should take the default value, pass

""

./export_tflite.sh   

Example:

./export_tflite.sh "" my_processing_dir my_models

Fine-Tuning

Fine-tuning is used when the alphabet is staying the same, since the alphabet stays the same when trying to add more German text corpora it can be used.

Consult the Fine-Tuning DeepSpeech Documentation

nohup ./fine_tuning_model.sh  &

Example:

nohup ./fine_tuning_model.sh german-speech-corpus/my_own_corpus &

Parameters:

./fine_tuning_model.sh      

• The default alphabet is

  data/alphabet.txt

  .
• The default DeepSpeech processing data is saved into

  deepspeech_processing

  .
• The scorer is taken from

  german-text-corpus/kenlm.scorer

  .
• The model is saved under the directory

  models

  .
• CUDA options:

  • -load_cudnn

    is not passed by default. If you don’t have a CUDA compatible GPU and the training was executed on a GPU (which is the default) pass

    --load_cudnn

    to allow training on the CPU. NOTE: This requires also the Python

    tensorflow-gpu

    dependency, although no GPU is available.

  • --train_cudnn

    is not passed by default. It must be passed if the existing model was trained on a CUDA GPU and it should be continued on a CUDA GPU.

Example:

./fine_tuning_model.sh german-speech-corpus/my_own_corpus "" "" "" "" "" "" --train_cudnn

Transfer Learning

Transfer learning must be used if the alphabet is changed, e.g. when using English and adding the German letters. Transfer learning is using an existing neural network, dropping one or more layers of this neural network and recalculated the weights with the new training data.

Consult the Transfer Learning DeepSpeech Documentation

Example English to German

• The English DeepSpeech model is used as base model. The checkpoints from the DeepSpeech English model must be us. E.g.

  deepspeech-0.9.3-checkpoint.tar.gz

  must be downloaded and extracted to

  /deepspeech-0.9.3-checkpoint

  .
• For the German language the language model and scorer must be build (this was done already above).
• Call

  transfer_model.sh

  :

nohup ./transfer_model.sh german-speech-corpus/data_tuda+voxforge+mozilla+swc+mailabs deepspeech-0.9.3-checkpoint &

NOTE: The checkpoints should be from the same DeepSpeech version to perform transfer learning.

Parameters:

• The default alphabet is

  data/alphabet.txt

  .
• The default DeepSpeech processing data is saved into

  deepspeech_transfer_processing

  .
• The scorer is taken from

  german-text-corpus/kenlm.scorer

  .
• The model is saved under the directory

  transfer_models

  .
• The drop layer parameter is 1.
• CUDA options:

  • -load_cudnn

    is not passed by default. If you don’t have a CUDA compatible GPU and the training was executed on a GPU (which is the default) pass

    --load_cudnn

    to allow training on the CPU. NOTE: This requires also the Python

    tensorflow-gpu

    dependency, although no GPU is available.

  • --train_cudnn

    is not passed by default. It must be passed if the existing model was trained on a CUDA GPU and it should be continued on a CUDA GPU.

It is possible to modify these parameters. If a parameter should take the default value, pass

""

nohup ./transfer_model.sh         &

Example:

nohup ./transfer_model.sh german-speech-corpus/data_tuda+voxforge+mozilla+swc+mailabs deepspeech-0.9.3-checkpoint "" "" "" my_models 2 --train_cudnn &

Hyper-Parameter Optimization

The learning rate can be tested with the script hyperparameter_optimization.sh.

NOTE: The drop out rate parameter is not tested in this script.

Execute it with:

nohup ./hyperparameter_optimization.sh  &

Results

Some WER (word error rate) results from our findings.

• Mozilla 79.7%
• Voxforge 72.1%
• Tuda-De 26.8%
• Tuda-De+Mozilla 57.3%
• Tuda-De+Voxforge 15.1%
• Tuda-De+Voxforge+Mozilla 21.5%

NOTE: Refer our paper for more information.

Trained Models

The DeepSpeech model can be directly re-trained on new datasets. The required dependencies are available at:

1. v0.5.0

This model is trained on DeepSpeech v0.5.0 with **Mozillav3+Voxforge+Tuda-De**_ (please refer the paper for more details) https://drive.google.com/drive/folders/1nG6xii2FP6PPqmcp4KtNVvUADXxEeakk?usp=sharing

https://drive.google.com/file/d/1VN1xPH0JQNKK6DiSVgyQ4STFyDY_rle3/view

2. v0.6.0

This model is trained on DeepSpeech v0.6.0 with **Mozillav4+Voxforge+Tuda-De+MAILABS(454+57+184+233h=928h)**_

https://drive.google.com/drive/folders/1BKblYaSLnwwkvVOQTQ5roOeN0SuQm8qr?usp=sharing

3. v0.7.4

This model is trained on DeepSpeech v0.7.4 using pre-trained English model released by Mozilla **English+Mozillav5+MAILABS+Tuda-De+Voxforge (1700+750+233+184+57h=2924h)**_

https://drive.google.com/drive/folders/1PFSIdmi4Ge8EB75cYh2nfYOXlCIgiMEL?usp=sharing

4. v0.9.0

This model is trained on DeepSpeech v0.9.0 using pre-trained English model released by Mozilla **English+Mozillav5+SWC+MAILABS+Tuda-De+Voxforge (1700+750+248+233+184+57h=3172h)**_

Thanks to Karsten Ohme for providing the TFLite model.

Link: https://drive.google.com/drive/folders/1L7ILB-TMmzL8IDYi_GW8YixAoYWjDMn1?usp=sharing

Why being SHY to STAR the repository, if you use the resources? :D

Acknowledgments

• Prof. Dr.-Ing. Torsten Zesch - Co-Author
• Dipl.-Ling. Andrea Horbach
• Matthias

References

If you use our findings/scripts in your academic work, please cite:

@inproceedings{agarwal-zesch-2019-german,
    author = "Aashish Agarwal and Torsten Zesch",
    title = "German End-to-end Speech Recognition based on DeepSpeech",
    booktitle = "Preliminary proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers",
    year = "2019",
    address = "Erlangen, Germany",
    publisher = "German Society for Computational Linguistics \& Language Technology",
    pages = "111--119"
}