Video Preprocessing

This repository provides tools for preprocessing videos for TaiChi, VoxCeleb and UvaNemo dataset used in paper.

Dowloading videos and cropping according to precomputed bounding boxes

1) Instal requirments:

pip install -r requirements.txt

2) Load youtube-dl:

wget https://yt-dl.org/downloads/latest/youtube-dl -O youtube-dl
chmod a+rx youtube-dl

3) Run script to download videos, there are 2 formats that can be used for storing videos one is .mp4 and another is folder with .png images. While .png images occupy significantly more space, the format is loss-less and have better i/o performance when training.

Taichi

python load_videos.py --metadata taichi-metadata.csv --format .mp4 --out_folder taichi --workers 8

VoxCeleb

python load_videos.py --metadata vox-metadata.csv --format .mp4 --out_folder vox --workers 8

UvaNemo Since videos is not avaliable on youtube you have to download videos from official website, and run:

python load_videos.py --metadata nemo-metadata.csv --format .mp4 --out_folder nemo --workers 8 --video_folder path/to/original/videos

Preprocessing VoxCeleb dataset

If you need to change cropping strategy for VoxCeleb dataset or produce new bounding box annotations folow these steps:

1) Load vox-celeb1(vox-celeb2) annotations:

wget www.robots.ox.ac.uk/~vgg/data/voxceleb/data/vox1_test_txt.zip
unzip vox1_test_txt.zip

wget www.robots.ox.ac.uk/~vgg/data/voxceleb/data/vox1_dev_txt.zip
unzip vox1_dev_txt.zip

wget www.robots.ox.ac.uk/~vgg/data/voxceleb/data/vox2_test_txt.zip
unzip vox2_test_txt.zip

wget www.robots.ox.ac.uk/~vgg/data/voxceleb/data/vox2_dev_txt.zip
unzip vox2_dev_txt.zip

2) Load youtube-dl:

wget https://yt-dl.org/downloads/latest/youtube-dl -O youtube-dl
chmod a+rx youtube-dl

3) Install face-alignment library:

git clone https://github.com/1adrianb/face-alignment
cd face-alignment
pip install -r requirements.txt
python setup.py install

4) Install ffmpeg

sudo apt-get install ffmpeg

5) Run preprocessing (assuming 8 gpu, and 5 workers per gpu).

python crop_vox.py --workers 40 --device_ids 0,1,2,3,4,5,6,7 --format .mp4 --dataset_version 2
python crop_vox.py --workers 40 --device_ids 0,1,2,3,4,5,6,7 --format .mp4 --dataset_version 1 --data_range 10000-11252

Preprocessing TaiChi dataset

If you need to change cropping strategy for TaiChi dataset or produce new bounding box annotations folow these steps:

1) Download videos based on annotations:

python load_videos.py --metadata taichi-metadata.csv --format .mp4 --out_folder taichi --workers 8 --video_folder youtube-taichi --no_crop

2) Install mask-rcnn benchmark. Follow the instalation guide https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/INSTALL.md

3) Load youtube-dl:

wget https://yt-dl.org/downloads/latest/youtube-dl -O youtube-dl
chmod a+rx youtube-dl

4) Run preprocessing (assuming 8 gpu, and 5 workers per gpu).

python crop_taichi.py --workers 40 --device_ids 0,1,2,3,4,5,6,7 --format .mp4

Preprocessing Nemo dataset

If you need to change cropping strategy for Nemo dataset or produce new bounding box annotations folow these steps:

1) Install face-alignment library:

git clone https://github.com/1adrianb/face-alignment
cd face-alignment
pip install -r requirements.txt
python setup.py install

2) Download videos from official website, and run:

python crop_nemo.py --in_folder /path/to/videos --out_folder nemo --device_ids 0,1 --workers 8 --format .mp4

Additional notes

Citation:

@InProceedings{Siarohin_2019_NeurIPS,
  author={Siarohin, Aliaksandr and Lathuilière, Stéphane and Tulyakov, Sergey and Ricci, Elisa and Sebe, Nicu},
  title={First Order Motion Model for Image Animation},
  booktitle = {Conference on Neural Information Processing Systems (NeurIPS)},
  month = {December},
  year = {2019}
}