avatar-net

by LucasSheng

LucasSheng /avatar-net

Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration

137 Stars 32 Forks Last release: Not found 51 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration

This repository contains the code (in TensorFlow) for the paper:

Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
Lu Sheng, Ziyi Lin, Jing Shao, Xiaogang Wang
CVPR 2018

Overview

In this repository, we propose an efficient and effective Avatar-Net that enables visually plausible multi-scale transfer for arbitrary style in real-time. The key ingredient is a style decorator that makes up the content features by semantically aligned style features, which does not only holistically match their feature distributions but also preserve detailed style patterns in the decorated features. By embedding this module into an image reconstruction network that fuses multi-scale style abstractions, the Avatar-Net renders multi-scale stylization for any style image in one feed-forward pass.

teaser

Examples

image_results

Comparison with Prior Arts

  • The result by Avatar-Net receives concrete multi-scale style patterns (e.g. color distribution, brush strokes and circular patterns in candy image).
  • WCT distorts the brush strokes and circular patterns. AdaIN cannot even keep the color distribution, while Style-Swap fails in this example.

Execution Efficiency

|Method| Gatys et. al. | AdaIN | WCT | Style-Swap | Avatar-Net | | :---: | :---: | :---: | :---: | :---: | :---: | | 256x256 (sec) | 12.18 | 0.053 | 0.62 | 0.064 | 0.071 | | 512x512 (sec) | 43.25 | 0.11 | 0.93 | 0.23 | 0.28 |

  • Avatar-Net has a comparable executive time as AdaIN and GPU-accelerated Style-Swap, and is much faster than WCT and the optimization-based style transfer by Gatys et. al..
  • The reference methods and the proposed Avatar-Net are implemented on a same TensorFlow platform with a same VGG network as the backbone.

Dependencies

Download

  • The trained model of Avatar-Net can be downloaded through the Google Drive.
  • The training of our style transfer network requires pretrained VGG networks, and they can be obtained from the TF-Slim model repository. The encoding layers of Avatar-Net are also borrowed from pretrained VGG models.
  • MSCOCO dataset is applied for the training of the proposed image reconstruction network.

Usage

Basic Usage

Simply use the bash file

./scripts/evaluate_style_transfer.sh
to apply Avatar-Net to all content images in
CONTENT_DIR
from any style image in
STYLE_DIR
. For example,
bash ./scripts/evaluate_style_transfer.sh gpu_id CONTENT_DIR STYLE_DIR EVAL_DIR 
  • gpu_id
    : the mounted GPU ID for the TensorFlow session.
  • CONTENT_DIR
    : the directory of the content images. It can be
    ./data/contents/images
    for multiple exemplar content images, or
    ./data/contents/sequences
    for an exemplar content video.
  • STYLE_DIR
    : the directory of the style images. It can be
    ./data/styles
    for multiple exemplar style images.
  • EVAL_DIR
    : the output directory. It contains multiple subdirectories named after the names of the style images.

More detailed evaluation options can be found in

evaluate_style_transfer.py
, such as
python evaluate_style_transfer.py

Configuration

The detailed configuration of Avatar-Net is listed in

configs/AvatarNet.yml
, including the training specifications and network hyper-parameters. The style decorator has three options:
  • patch_size
    : the patch size for the normalized cross-correlation, in default is
    5
    .
  • style_coding
    : the projection and reconstruction method, either
    ZCA
    or
    AdaIN
    .
  • style_interp
    : interpolation option between the transferred features and the content features, either
    normalized
    or
    biased
    .

The style transfer is actually performed in

AvatarNet.transfer_styles(self, inputs, styles, inter_weight, intra_weights)
, in which
  • inputs
    : the content images.
  • styles
    : a list of style images (
    len(styles)
    > 2 for multiple style interpolation).
  • inter_weight
    : the weight balancing the style and content images.
  • intra_weights
    : a list of weights balancing the effects from different styles.

Users may modify the evaluation script for multiple style interpolation or content-style trade-off.

Training

  1. Download MSCOCO datasets and transfer the raw images into
    tfexamples
    , according to the python script
    ./datasets/convert_mscoco_to_tfexamples.py
    .
  2. Use
    bash ./scripts/train_image_reconstruction.sh gpu_id DATASET_DIR MODEL_DIR
    to start training with default hyper-parameters.
    gpu_id
    is the mounted GPU for the applied Tensorflow session. Replace
    DATASET_DIR
    with the path to MSCOCO training images and
    MODEL_DIR
    to Avatar-Net model directory.

Citation

If you find this code useful for your research, please cite the paper:

Lu Sheng, Ziyi Lin, Jing Shao and Xiaogang Wang, "Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration", in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. [Arxiv]

@inproceedings{sheng2018avatar,
    Title = {Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration},
    author = {Sheng, Lu and Lin, Ziyi and Shao, Jing and Wang, Xiaogang},
    Booktitle = {Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on},
    pages={1--9},
    year={2018}
}

Acknowledgement

This project is inspired by many style-agnostic style transfer methods, including AdaIN, WCT and Style-Swap, both from their papers and codes.

Contact

If you have any questions or suggestions about this paper, feel free to contact me ([email protected])

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.