Need help with Autoregressive-Predictive-Coding?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

iamyuanchung
147 Stars 27 Forks 7 Commits 1 Opened issues

Description

Autoregressive Predictive Coding: An unsupervised autoregressive model for speech representation learning

Services available

!
?

Need anything else?

Contributors list

# 247,611
C++
C
machine...
self-su...
7 commits

Autoregressive Predictive Coding

This repository contains the official implementation (in PyTorch) of Autoregressive Predictive Coding (APC) proposed in An Unsupervised Autoregressive Model for Speech Representation Learning.

APC is a speech feature extractor trained on a large amount of unlabeled data. With an unsupervised, autoregressive training objective, representations learned by APC not only capture general acoustic characteristics such as speaker and phone information from the speech signals, but are also highly accessible to downstream models--our experimental results on phone classification show that a linear classifier taking the APC representations as the input features significantly outperforms a multi-layer percepron using the surface features.

Dependencies

  • Python 3.5
  • PyTorch 1.0

Dataset

In the paper, we used the train-clean-360 split from the LibriSpeech corpus for training the APC models, and the dev-clean split for keeping track of the training loss. We used the log Mel spectrograms, which were generated by running the Kaldi scripts, as the input acoustic features to the APC models. Of course you can generate the log Mel spectrograms yourself, but to help you better reproduce our results, here we provide the links to the data proprocessed by us that can be directly fed to the APC models. We also include other data splits that we did not use in the paper for you to explore, e.g., you can try training an APC model on a larger and nosier set (e.g., train-other-500) and see if it learns more robust speech representations. * train-clean-100 * train-clean-360 (used for training APC models in our paper) * train-other-500 * dev-clean (used for tracing the training loss)

Training APC

Below we will follow the paper and use train-clean-360 and dev-clean as demonstration. Once you have downloaded the data, unzip them by running:

bash
xz -d train-clean-360.xz
xz -d dev-clean.xz
Then, create a directory
librispeech_data/kaldi
and move the data into it:
bash
mkdir -p librispeech_data/kaldi
mv train-clean-360-hires-norm.blogmel librispeech_data/kaldi
mv dev-clean-hires-norm.blogmel librispeech_data/kaldi
Now we will have to transform the data into the format loadable by the PyTorch DataLoader. To do so, simply run: ```bash

Prepare the training set

python preparedata.py --librispeechfromkaldi librispeechdata/kaldi/train-clean-360-hires-norm.blogmel --savedir librispeechdata/preprocessed/train-clean-360-hires-norm.blogmel

Prepare the valication set

python preparedata.py --librispeechfromkaldi librispeechdata/kaldi/dev-clean-hires-norm.blogmel --savedir librispeechdata/preprocessed/dev-clean-hires-norm-blogmel ``

Once the program is done, you will see a directory
preprocessed/
inside
librispeech_data/` that contains all the preprocessed PyTorch tensors.

To train an APC model, simply run:

bash
python train_apc.py
By default, the trained models will be put in
logs/
. You can also use Tensorboard to trace the training progress. There are many other configurations you can try, check
train_apc.py
for more details--it is highly documented and should be self-explanatory.

Feature extraction

Once you have trained your APC model, you can use it to extract speech features from your target dataset. To do so, feed-forward the trained model on the target dataset and retrieve the extracted features by running:

bash
_, feats = model.forward(inputs, lengths)
feats
is a PyTorch tensor of shape (
num_layers
,
batch_size
,
seq_len
,
rnn_hidden_size
) where: -
num_layers
is the RNN depth of your APC model -
batch_size
is your inference batch size -
seq_len
is the maximum sequence length and is determined when you run
prepare_data.py
. By default this value is 1600. -
rnn_hidden_size
is the dimensionality of the RNN hidden unit.

As you can see,

feats
is essentially the RNN hidden states in an APC model. You can think of APC as a speech version of ELMo if you are familiar with it.

There are many ways to incorporate

feats
into your downstream task. One of the easiest way is to take only the outputs of the last RNN layer (i.e.,
feats[-1, :, :, :]
) as the input features to your downstream model, which is what we did in our paper. Feel free to explore other mechanisms.

Pre-trained models

We release the pre-trained models that were used to produce the numbers reported in the paper.

load_pretrained_model.py
provides a simple example of loading a pre-trained model. * n = 1 * n = 2 * n = 3 * n = 5 * n = 10 * n = 20

Reference

Please cite our paper(s) if you find this repository useful. This first paper proposes the APC objective, while the second paper applies it to speech recognition, speech translation, and speaker identification, and provides more systematic analysis on the learned representations. Cite both if you are kind enough!

@inproceedings{chung2019unsupervised,
  title = {An unsupervised autoregressive model for speech representation learning},
  author = {Chung, Yu-An and Hsu, Wei-Ning and Tang, Hao and Glass, James},
  booktitle = {Interspeech},
  year = {2019}
}
@inproceedings{chung2020generative,
  title = {Generative pre-training for speech with autoregressive predictive coding},
  author = {Chung, Yu-An and Glass, James},
  booktitle = {ICASSP},
  year = {2020}
}

Contact

Feel free to shoot me an email for any inquiries about the paper and this repository.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.