Need help with nonparaSeq2seqVC_code?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

jxzhanggg
175 Stars 45 Forks MIT License 55 Commits 10 Opened issues

Description

Implementation code of non-parallel sequence-to-sequence VC

Services available

!
?

Need anything else?

Contributors list

# 20,556
Keras
C++
Tensorf...
ml
17 commits

Non-parallel Seq2seq Voice Conversion

Implementation code of Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representations.

For audio samples, please visit our demo page.

The structure overview of the model

Dependencies

  • Python 3.6
  • PyTorch 1.0.1
  • CUDA 10.0

Data

It is recommended you download the VCTK and CMU-ARCTIC datasets.

Usage

Installation

Install Python dependencies.

$ pip install -r requirements.txt

Feature Extraction

Extract Mel-Spectrograms, Spectrograms and Phonemes

You can use

extract_features.py

Customize data reader

Write a snippet of code to walk through the dataset for generating list file for train, valid and test set.

Then you will need to modify the data reader to read your training data. The following are scripts you will need to modify.

For pre-training:

For fine-tuning:

Pre-train the model

Add correct paths to your local data, and run the bash script:

$ cd pre-train
$ bash run.sh

Run the inference code to generate audio samples on multi-speaker dataset. During inference, our model can be run on either TTS (using text inputs) or VC (using Mel-spectrogram inputs) mode.

$ python inference.py

Fine-tune the model

Fine-tune the model and generate audio samples on conversion pair. During inference, our model can be run on either TTS (using text inputs) or VC (using Mel-spectrogram inputs) mode.

$ cd fine-tune
$ bash run.sh

Training Time

On a single NVIDIA 1080 Ti GPU, with a batch size of 32, pre-training on VCTK takes approximately 64 hours of wall-clock time. Fine-tuning on two speakers (500 utterances each speaker) with a batch size of 8 takes approximately 6 hours of wall-clock time.

Citation

If you use this code, please cite: ```bibtex @article{zhangnonpara2020, author={Jing-Xuan {Zhang} and Zhen-Hua {Ling} and Li-Rong {Dai}}, journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing}, title={Non-Parallel Sequence-to-Sequence Voice Conversion with Disentangled Linguistic and Speaker Representations}, year={2020}, volume={28}, number={1}, pages={540-552}}

## Acknowledgements

Part of code was adapted from the following project:

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.