Rotate-and-Render

by Hangz-nju-cuhk

Hangz-nju-cuhk / Rotate-and-Render

Code for Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images (CVPR ...

261 Stars 56 Forks Last release: Not found Creative Commons Attribution 4.0 International 79 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images (CVPR 2020)

Hang Zhou*, Jihao Liu*, Ziwei Liu, Yu Liu, and Xiaogang Wang.

[Paper] [Video]

We propose a novel un-supervised framework that can synthesize photorealistic rotated faces using only single-view image collections in the wild. Our key insight is that rotating faces in the 3D space back and forth, and re-rendering them to the 2D plane can serve as a strong self-supervision.

Requirements

  • Python 3.6 is used. Basic requirements are listed in the 'requirements.txt'.
pip install -r requirements.txt
  • Install the Neural_Renderer following the instructions.

  • Download checkpoint and BFM model from ckptandbfm.zip, put it in

    3ddfa
    and unzip it. Our 3D models are borrowed from 3DDFA.

DEMO

Rotate faces with examples provided

  1. Download the checkpoint and put it in

    ./checkpoints/rs_model
    .
  2. Run a simple Rotate-and-Render demo, the inputs are stored at

    3ddfa/example
    .
  • Modify

    experiments/v100_test.sh
    , the
    --poses
    are the desired degrees (range -90 to 90), choose 0 as frontal face.
  • Run

    bash experiments/v100_test.sh
    , results will be saved at
    ./results/
    .

DEVELOP

Prepare your own dataset for testing and training.

Preprocessing

  1. Save the 3D params of human faces to
    3ddfa/results
    by 3ddfa.
    bash
    cd 3ddfa
    python inference.py --img_list example/file_list.txt --img_prefix example/Images --save_dir results
    cd ..
    

Data Preparation

Modify

class dataset_info()
inside
data/__ini__.py
, then prepare dataset according to the pattern of the existing example. You can add the information about a new dataset to each instance of the class.
  1. prefix
    The absolute path to the dataset.
  2. file_list
    The list of all images, the absolute path could be incorrect as it is defined in the
    prefix
  3. land_mark_list
    The list that stores all landmarks of all images.
  4. params_dir
    the path that stores all the 3D params processed before.
  5. dataset_names
    the dictionary that maps dataset NAMEs to their information. This is used in the parsers as
    --dataset NAME
    .
  6. folder_level
    the level of folders from the
    prefix
    to images (.jpgs). For example the folder_level is 2 if a image is stored as
    prefix/label/image.jpg
    .

Training and Inference

  • Modify

    experiments/train.sh
    and use
    bash experiments/train.sh
    for training on new datasets.
  • Visualization is supported during training with Tensorboard.

  • Please see the DEMO part for inference.

Details of the Models

We provide two models with trainers in this repo, namely

rotate
and
rotatespade
. The "rotatespade" model is an upgraded one which is different from that described in our paper. A conditional batchnorm module is added according to landmarks predicted from the 3D face. Our checkpoint is trained on this model. We have briefly described this model in our supplementary materials.

License and Citation

The usage of this software is under CC-BY-4.0.

@inproceedings{zhou2020rotate,
  title     = {Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images},
  author    = {Zhou, Hang and Liu, Jihao and Liu, Ziwei and Liu, Yu and Wang, Xiaogang},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2020},
}

Acknowledgement

  • The structure of this codebase is borrowed from SPADE.
  • The SyncBN module is used in our code.
  • We directly borrow the 3DDFA implementation for 3D reconstruction.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.