Need help with deepspeech-german?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

AASHISHAG
169 Stars 23 Forks Apache License 2.0 122 Commits 1 Opened issues

Description

Automatic Speech Recognition (ASR) - German

Services available

!
?

Need anything else?

Contributors list

Automatic Speech Recognition (ASR) - DeepSpeech German

This is the project for the paper German End-to-end Speech Recognition based on DeepSpeech published at KONVENS 2019.

This project aims to develop a working Speech to Text module using Mozilla DeepSpeech, which can be used for any Audio processing pipeline. Mozilla DeepSpeech Architecture is a state-of-the-art open-source automatic speech recognition (ASR) toolkit. DeepSpeech is using a model trained by machine learning techniques based on Baidu's Deep Speech research paper. Project DeepSpeech uses Google's TensorFlow to make the implementation easier.

DeepSpeech

Important Links:

Paper: https://www.researchgate.net/publication/336532830GermanEnd-to-endSpeechRecognitionbasedon_DeepSpeech

DeepSpeech-API: https://github.com/AASHISHAG/DeepSpeech-API

This Readme is written for DeepSpeech v0.9.3. Refer to Mozillla DeepSpeech for latest updates.

Contents

  1. Requirements
  2. Speech Corpus
  3. Language Model
  4. Training
  5. Hyper-Parameter Optimization
  6. Results
  7. Trained Models
  8. Acknowledgments
  9. References

Requirements

The instruction are focusing on Linux. It might be possible to use also MacOS and Windows as long as a shell environment can be provided, however some instructions and scripts might have to be adjusted then.

DeepSpeech
and
KenLM
were added as sub modules. To fetch the sub modules execute:
git pull --recurse-submodules

Developer Information

To update the used DeepSpeech version checkout in the

DeepSpeech
submodule the corresponding tag:
git checkout tags/v0.9.3

The same applies to KenLM.

Installing Python bindings

The DeepSpeech tools require still TensorFlow 1.15, which is only supported up to Python 3.7.

pyenv
will be used to set up a dedicated Python version.

git clone https://github.com/pyenv/pyenv.git $HOME/.pyenv

Follow the instruction on the pyenv page, for Ubuntu Desktop Add to your

~/.bashrc
:
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bash_profile

Log out or run:

source ~/.bashrc

virtualenv
will be used to provide a separate Python environment.

pyenv install 3.7.9
sudo pip3 install virtualenv
virtualenv --python=$HOME/.pyenv/versions/3.7.9/bin/python3.7 python-environments
source python-environments/bin/activate
pip3 install -r python_requirements.txt

NOTE: While writing this 2 issues were fixed in audiomate and a GitHub patched version was used. This can be updated once the upstream audiomate has released a new version.

Installing Linux dependencies

The necessary Linux dependencies can be found in

linux_requirements
.
xargs -a linux_requirements.txt sudo apt-get install

Speech Corpus

Download the corpus

For the speech corpus download the script

download_speech_corpus.py
is used and it will take the above URLs.

source python-environments/bin/activate
python pre-processing/download_speech_corpus.py --tuda --cv --swc --voxforge --mailabs

NOTE: The python module of

audiomate
which is used for the download process is using some fixed URLs. These URLs can be outdated depending on last release of
audiomate
. `download
speechcorpus.py
is patching these URLS, but in case a new speech corpus is released 
the URLs in the script must be updated again or passed as arguments
<speech
corpus>url
, e.g.
--tuda
url `._

It is also possible to leave out some corpus and specify just a subset to download. The default target directory is the current directory, i.e. project root but could be set with

--target_path
. The path used in this documentation must be adjusted then accordingly.

The tool will display after a while a download progress indicator.

The downloads will be found in the folder

tuda
,
mozilla
,
voxforge
,
swc
and
mailabs
. If a download failed the incomplete file(s) must be removed manually.

Prepare the Audio Data

The output of this step is a CSV format with references to the audio and corresponding text split into train, dev and test data for the TensorFlow network.

Convert to UTF-8:

pre-processing/run_to_utf_8.sh

The

pre-processing/prepare_data
script imports the downloaded corpora. It expects the corpus option followed by the corpus directory followed by the destination directory. Look into the script source to find all possible corpus options. It is possible to train a subset (i.e. by just specifying one of the options) or all of the speech corpora.

Execute the following commands in the root directory of this project after running:

source python-environments/bin/activate

NOTE: If the filename or folders of the downloaded speech corpora are changing the paths used in this section must be updated, too.

Some Examples:

1. Tuda-De

python pre-processing/prepare_data.py --tuda tuda/german-speechdata-package-v3 german-speech-corpus/data_tuda

2. Mozilla

python pre-processing/prepare_data.py --cv mozilla/cv-corpus-6.1-2020-12-11/de german-speech-corpus/data_mozilla

3. Voxforge

python pre-processing/prepare_data.py --voxforge voxforge german-speech-corpus/data_voxforge

4. Voxforge + Tuda-De

python pre-processing/prepare_data.py --voxforge voxforge --tuda tuda/german-speechdata-package-v2 german-speech-corpus/data_tuda+voxforge

5. Voxforge + Tuda-De + Mozilla

python pre-processing/prepare_data.py --cv mozilla/cv-corpus-6.1-2020-12-11/de/clips --voxforge voxforge --tuda tuda/german-speechdata-package-v3 german-speech-corpus/data_tuda+voxforge+mozilla

Language Model

We used KenLM toolkit to train a 3-gram language model. It is Language Model inference code by Kenneth Heafield Consult the repository for any information regarding the compilation in case of errors.

  • Installation
cd kenlm
mkdir -p build
cd build
cmake ..
make -j `nproc`

nproc
defines the number of threads to use.

Text Corpus

We used an open-source German Speech Corpus released by University of Hamburg.

Download the data:

mkdir german-text-corpus
cd german-text-corpus
wget http://ltdata1.informatik.uni-hamburg.de/kaldi_tuda_de/German_sentences_8mil_filtered_maryfied.txt.gz
gzip -d German_sentences_8mil_filtered_maryfied.txt.gz

Pre-process the data

Execute the following commands in the root directory of this project.

source python-environments/bin/activate
python pre-processing/prepare_vocab.py german-text-corpus/German_sentences_8mil_filtered_maryfied.txt german-text-corpus/clean_vocab.txt

Build the Language Model

This step will create a

trie
data structure to describe the used vocabulary for the speech recognition. This file is termed as "scorer" in the DeepSpeech terminology. The script
create_language_model.sh
is used for this.

The script will use the compiled KenLM binaries and download a native DeepSpeechClient. The native DeepSpeech client is using the "cpu" architecture. For a different architecture pass as additional parameter the architecture. See the

--arch
parameter in
python DeepSpeech/util/taskcluster.py --help
for details.

The mandatory

top_k
parameter in the script is set to 500000 which corresponds to > 98.43 % of all words in the downloaded text corpus.

The

default_alpha
and
default_beta
parameters are taken from Scorer documentation and are also the defaults in the
DeepSpeech.py
. DeepSpeech is offering the
lm_optimizer.py
scripts to find better values (in case there would be different ones)._
./create_language_model.sh

NOTE: A different scorer might be useful later for the efficient speech recognition in case of a special language domain. Then an own text corpus should be used, and the script source can be used as a template for this scorer and be executed separately.

Training

NOTE: The training can be accelerated by using CUDA. Follow the GPU Installation Instructions for TensorFlow. At the time of writing also on Ubuntu 20.04 the toolchain for Ubuntu 18.04 has to be used.

If CUDA is used the GPU package of TensorFlow has to be installed:

source python-environments/bin/activate
pip uninstall tensorflow
pip install tensorflow-gpu==1.15.4
nohup ./train_model.sh  &

Example:

nohup ./train_model.sh german-speech-corpus/data_tuda+voxforge+mozilla &

Parameters:

  • The default scorer is
    german-text-corpus/kenlm.scorer
    .
  • The default alphabet is
    data/alphabet.txt
    .
  • The default DeepSpeech processing data is stored under
    deepspeech_processing
    .
  • The model is stored under the directory
    models
    .

It is possible to modify these parameters. If a parameter should take the default value, pass

""
.
./train_model.sh     

Example:

./train_model.sh german-speech-corpus/data_tuda+voxforge+mozilla "" "" my_processing_dir my_models

NOTE: The augmentation parameters are not used and might be interesting for improving the results.

NOTE: Increase the `exportmodelversion

option in
trainmodel.sh` when a new model is trained._

NOTE: In case a

Not enough time for target transition sequence (required: 171, available: 0)
is thrown, the currently only known fix is to edit the file `DeepSpeech/training/deepspeechtraining/train.py

and
add
, ignorelongeroutputsthaninputs=True
to the call to
tfv1.nn.ctcloss`.

TFLite Model Generation

The TensorFLow Lite model suitable for resource restricted devices (embedded, Raspberry Pi 4, ...) can be generated with:

./export_tflite.sh

Parameters:

  • The default alphabet is
    data/alphabet.txt
    .
  • The default DeepSpeech processing data is taken from
    deepspeech_processing
    .
  • The model is stored under the directory
    models
    .

It is possible to modify these parameters. If a parameter should take the default value, pass

""
.
./export_tflite.sh   

Example:

./export_tflite.sh "" my_processing_dir my_models

Fine-Tuning

Fine-tuning is used when the alphabet is staying the same, since the alphabet stays the same when trying to add more German text corpora it can be used.

Consult the Fine-Tuning DeepSpeech Documentation

nohup ./fine_tuning_model.sh  &

Example:

nohup ./fine_tuning_model.sh german-speech-corpus/my_own_corpus &

Parameters:

./fine_tuning_model.sh      
  • The default alphabet is
    data/alphabet.txt
    .
  • The default DeepSpeech processing data is saved into
    deepspeech_processing
    .
  • The scorer is taken from
    german-text-corpus/kenlm.scorer
    .
  • The model is saved under the directory
    models
    .
  • CUDA options:
    • -load_cudnn
      is not passed by default. If you don’t have a CUDA compatible GPU and the training was executed on a GPU (which is the default) pass
      --load_cudnn
      to allow training on the CPU. NOTE: This requires also the Python
      tensorflow-gpu
      dependency, although no GPU is available.
    • --train_cudnn
      is not passed by default. It must be passed if the existing model was trained on a CUDA GPU and it should be continued on a CUDA GPU.

Example:

./fine_tuning_model.sh german-speech-corpus/my_own_corpus "" "" "" "" "" "" --train_cudnn

Transfer Learning

Transfer learning must be used if the alphabet is changed, e.g. when using English and adding the German letters. Transfer learning is using an existing neural network, dropping one or more layers of this neural network and recalculated the weights with the new training data.

Consult the Transfer Learning DeepSpeech Documentation

Example English to German

  • The English DeepSpeech model is used as base model. The checkpoints from the DeepSpeech English model must be us. E.g.
    deepspeech-0.9.3-checkpoint.tar.gz
    must be downloaded and extracted to
    /deepspeech-0.9.3-checkpoint
    .
  • For the German language the language model and scorer must be build (this was done already above).
  • Call
    transfer_model.sh
    :
nohup ./transfer_model.sh german-speech-corpus/data_tuda+voxforge+mozilla+swc+mailabs deepspeech-0.9.3-checkpoint &

NOTE: The checkpoints should be from the same DeepSpeech version to perform transfer learning.

Parameters:

  • The default alphabet is
    data/alphabet.txt
    .
  • The default DeepSpeech processing data is saved into
    deepspeech_transfer_processing
    .
  • The scorer is taken from
    german-text-corpus/kenlm.scorer
    .
  • The model is saved under the directory
    transfer_models
    .
  • The drop layer parameter is 1.
  • CUDA options:
    • -load_cudnn
      is not passed by default. If you don’t have a CUDA compatible GPU and the training was executed on a GPU (which is the default) pass
      --load_cudnn
      to allow training on the CPU. NOTE: This requires also the Python
      tensorflow-gpu
      dependency, although no GPU is available.
    • --train_cudnn
      is not passed by default. It must be passed if the existing model was trained on a CUDA GPU and it should be continued on a CUDA GPU.

It is possible to modify these parameters. If a parameter should take the default value, pass

""
.
nohup ./transfer_model.sh         &

Example:

nohup ./transfer_model.sh german-speech-corpus/data_tuda+voxforge+mozilla+swc+mailabs deepspeech-0.9.3-checkpoint "" "" "" my_models 2 --train_cudnn &

Hyper-Parameter Optimization

The learning rate can be tested with the script hyperparameter_optimization.sh.

NOTE: The drop out rate parameter is not tested in this script.

Execute it with:

nohup ./hyperparameter_optimization.sh  &

Results

Some WER (word error rate) results from our findings.

  • Mozilla 79.7%
  • Voxforge 72.1%
  • Tuda-De 26.8%
  • Tuda-De+Mozilla 57.3%
  • Tuda-De+Voxforge 15.1%
  • Tuda-De+Voxforge+Mozilla 21.5%

NOTE: Refer our paper for more information.

Trained Models

The DeepSpeech model can be directly re-trained on new datasets. The required dependencies are available at:

1. v0.5.0

This model is trained on DeepSpeech v0.5.0 with **Mozillav3+Voxforge+Tuda-De**_ (please refer the paper for more details) https://drive.google.com/drive/folders/1nG6xii2FP6PPqmcp4KtNVvUADXxEeakk?usp=sharing

https://drive.google.com/file/d/1VN1xPH0JQNKK6DiSVgyQ4STFyDY_rle3/view

2. v0.6.0

This model is trained on DeepSpeech v0.6.0 with **Mozillav4+Voxforge+Tuda-De+MAILABS(454+57+184+233h=928h)**_

https://drive.google.com/drive/folders/1BKblYaSLnwwkvVOQTQ5roOeN0SuQm8qr?usp=sharing

3. v0.7.4

This model is trained on DeepSpeech v0.7.4 using pre-trained English model released by Mozilla **English+Mozillav5+MAILABS+Tuda-De+Voxforge (1700+750+233+184+57h=2924h)**_

https://drive.google.com/drive/folders/1PFSIdmi4Ge8EB75cYh2nfYOXlCIgiMEL?usp=sharing

4. v0.9.0

This model is trained on DeepSpeech v0.9.0 using pre-trained English model released by Mozilla **English+Mozillav5+SWC+MAILABS+Tuda-De+Voxforge (1700+750+248+233+184+57h=3172h)**_

Thanks to Karsten Ohme for providing the TFLite model.

Link: https://drive.google.com/drive/folders/1L7ILB-TMmzL8IDYi_GW8YixAoYWjDMn1?usp=sharing

Why being SHY to STAR the repository, if you use the resources? :D

Acknowledgments

References

If you use our findings/scripts in your academic work, please cite:

@inproceedings{agarwal-zesch-2019-german,
    author = "Aashish Agarwal and Torsten Zesch",
    title = "German End-to-end Speech Recognition based on DeepSpeech",
    booktitle = "Preliminary proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers",
    year = "2019",
    address = "Erlangen, Germany",
    publisher = "German Society for Computational Linguistics \& Language Technology",
    pages = "111--119"
}
<!-- An open-access Arxiv preprint is available here: -->

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.