Github url

CycleGAN

by junyanz

junyanz /CycleGAN

Software that can generate photos from paintings, turn horses into zebras, perform style transfer,...

9.8K Stars 1.7K Forks Last release: Not found Other 96 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

CycleGAN

PyTorch | project page | paper

Torch implementation for learning an image-to-image translation (i.e. pix2pix) without input-output pairs, for example:

Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Jun-Yan Zhu*, Taesung Park*, Phillip Isola, Alexei A. Efros
Berkeley AI Research Lab, UC Berkeley
In ICCV 2017. (* equal contributions)

This package includes CycleGAN, pix2pix, as well as other methods like BiGAN/ALI and Apple's paper S+U learning. The code was written by Jun-Yan Zhu and Taesung Park. Update: Please check out PyTorch implementation for CycleGAN and pix2pix. The PyTorch version is under active development and can produce results comparable or better than this Torch version.

Other implementations:

[Tensorflow] (by Harry Yang),[Tensorflow] (by Archit Rathore),[Tensorflow] (by Van Huy),[Tensorflow] (by Xiaowei Hu), [Tensorflow-simple] (by Zhenliang He),[TensorLayer] (by luoxier),[Chainer] (by Yanghua Jin),[Minimal PyTorch] (by yunjey),[Mxnet] (by Ldpe2G),[lasagne/Keras] (by tjwei), [Keras] (by Simon Karlsson)

Applications

Monet Paintings to Photos

Collection Style Transfer

Object Transfiguration

Season Transfer

Photo Enhancement: Narrow depth of field

Prerequisites

  • Linux or OSX
  • NVIDIA GPU + CUDA CuDNN (CPU mode and CUDA without CuDNN may work with minimal modification, but untested)
  • For MAC users, you need the Linux/GNU commands
    gfind
    and
    gwc
    , which can be installed with
    brew install findutils coreutils
    .

Getting Started

Installation

bash luarocks install nngraph luarocks install class luarocks install https://raw.githubusercontent.com/szym/display/master/display-scm-0.rockspec
  • Clone this repo:
    bash git clone https://github.com/junyanz/CycleGAN cd CycleGAN

Apply a Pre-trained Model

  • Download the test photos (taken by Alexei Efros):
    bash ./datasets/download\_dataset.sh ae\_photos
  • Download the pre-trained model
    style\_cezanne
    (For CPU model, use
    style\_cezanne\_cpu
    ):
    bash ./pretrained\_models/download\_model.sh style\_cezanne
  • Now, let's generate Paul Cézanne style images:
    DATA\_ROOT=./datasets/ae\_photos name=style\_cezanne\_pretrained model=one\_direction\_test phase=test loadSize=256 fineSize=256 resize\_or\_crop="scale\_width" th test.lua
    The test results will be saved to
    ./results/style\_cezanne\_pretrained/latest\_test/index.html
    . Please refer to Model Zoo for more pre-trained models.
    ./examples/test\_vangogh\_style\_on\_ae\_photos.sh
    is an example script that downloads the pretrained Van Gogh style network and runs it on Efros's photos.

Train

  • Download a dataset (e.g. zebra and horse images from ImageNet):
    bash bash ./datasets/download\_dataset.sh horse2zebra
  • Train a model:
    bash DATA\_ROOT=./datasets/horse2zebra name=horse2zebra\_model th train.lua
  • (CPU only) The same training command without using a GPU or CUDNN. Setting the environment variables
    gpu=0 cudnn=0
    forces CPU only
    bash DATA\_ROOT=./datasets/horse2zebra name=horse2zebra\_model gpu=0 cudnn=0 th train.lua
  • (Optionally) start the display server to view results as the model trains. (See Display UI for more details):
    bash th -ldisplay.start 8000 0.0.0.0

Test

  • Finally, test the model:
    bash DATA\_ROOT=./datasets/horse2zebra name=horse2zebra\_model phase=test th test.lua
    The test results will be saved to an HTML file here:
    ./results/horse2zebra\_model/latest\_test/index.html
    .

Model Zoo

Download the pre-trained models with the following script. The model will be saved to

./checkpoints/model\_name/latest\_net\_G.t7

.

bash bash ./pretrained\_models/download\_model.sh model\_name
  • orange2apple
    (orange -> apple) and
    apple2orange
    : trained on ImageNet categories
    apple
    and
    orange
    . -
    horse2zebra
    (horse -> zebra) and
    zebra2horse
    (zebra -> horse): trained on ImageNet categories
    horse
    and
    zebra
    . -
    style\_monet
    (landscape photo -> Monet painting style),
    style\_vangogh
    (landscape photo -> Van Gogh painting style),
    style\_ukiyoe
    (landscape photo -> Ukiyo-e painting style),
    style\_cezanne
    (landscape photo -> Cezanne painting style): trained on paintings and Flickr landscape photos. -
    monet2photo
    (Monet paintings -> real landscape): trained on paintings and Flickr landscape photographs. -
    cityscapes\_photo2label
    (street scene -> label) and
    cityscapes\_label2photo
    (label -> street scene): trained on the Cityscapes dataset. -
    map2sat
    (map -> aerial photo) and
    sat2map
    (aerial photo -> map): trained on Google maps. -
    iphone2dslr\_flower
    (iPhone photos of flowers -> DSLR photos of flowers): trained on Flickr photos.

CPU models can be downloaded using:

bash bash pretrained\_models/download\_model.sh <name>_cpu
</name>

, where

<name></name>

can be

horse2zebra

,

style\_monet

, etc. You just need to append

\_cpu

to the target model.

Training and Test Details

To train a model,

bash DATA\_ROOT=/path/to/data/ name=expt\_name th train.lua

Models are saved to

./checkpoints/expt\_name

(can be changed by passing

checkpoint\_dir=your\_dir

in train.lua).
See

opt\_train

in

options.lua

for additional training options.

To test the model,

bash DATA\_ROOT=/path/to/data/ name=expt\_name phase=test th test.lua

This will run the model named

expt\_name

in both directions on all images in

/path/to/data/testA

and

/path/to/data/testB

. A webpage with result images will be saved to

./results/expt\_name

(can be changed by passing

results\_dir=your\_dir

in test.lua).
See

opt\_test

in

options.lua

for additional test options. Please use

model=one\_direction\_test

if you only would like to generate outputs of the trained network in only one direction, and specify

which\_direction=AtoB

or

which\_direction=BtoA

to set the direction.

There are other options that can be used. For example, you can specify

resize\_or\_crop=crop

option to avoid resizing the image to squares. This is indeed how we trained GTA2Cityscapes model in the projet webpage and Cycada model. We prepared the images at 1024px resolution, and used

resize\_or\_crop=crop fineSize=360

to work with the cropped images of size 360x360. We also used

lambda\_identity=1.0

.

Datasets

Download the datasets using the following script. Many of the datasets were collected by other researchers. Please cite their papers if you use the data.

bash bash ./datasets/download\_dataset.sh dataset\_name
  • facades
    : 400 images from the CMP Facades dataset. [Citation] -
    cityscapes
    : 2975 images from the Cityscapes training set. [Citation]. Note: Due to license issue, we do not host the dataset on our repo. Please download the dataset directly from the Cityscapes webpage. Please refer to
    ./datasets/prepare\_cityscapes\_dataset.py
    for more detail. -
    maps
    : 1096 training images scraped from Google Maps. -
    horse2zebra
    : 939 horse images and 1177 zebra images downloaded from ImageNet using the keywords
    wild horse
    and
    zebra
  • apple2orange
    : 996 apple images and 1020 orange images downloaded from ImageNet using the keywords
    apple
    and
    navel orange
    . -
    summer2winter\_yosemite
    : 1273 summer Yosemite images and 854 winter Yosemite images were downloaded using Flickr API. See more details in our paper. -
    monet2photo
    ,
    vangogh2photo
    ,
    ukiyoe2photo
    ,
    cezanne2photo
    : The art images were downloaded from Wikiart. The real photos are downloaded from Flickr using the combination of the tags landscape and landscapephotography. The training set size of each class is Monet:1074, Cezanne:584, Van Gogh:401, Ukiyo-e:1433, Photographs:6853. -
    iphone2dslr\_flower
    : both classes of images were downloaded from Flickr. The training set size of each class is iPhone:1813, DSLR:3316. See more details in our paper.

    Display UI

Optionally, for displaying images during training and test, use the display package.

  • Install it with:
    luarocks install https://raw.githubusercontent.com/szym/display/master/display-scm-0.rockspec
  • Then start the server with:
    th -ldisplay.start
  • Open this URL in your browser: http://localhost:8000

By default, the server listens on localhost. Pass

0.0.0.0

to allow external connections on any interface:

bash th -ldisplay.start 8000 0.0.0.0

Then open

http://(hostname):(port)/

in your browser to load the remote desktop.

Setup Training and Test data

To train CycleGAN model on your own datasets, you need to create a data folder with two subdirectories

trainA

and

trainB

that contain images from domain A and B. You can test your model on your training set by setting

phase='train'

in

test.lua

. You can also create subdirectories

testA

and

testB

if you have test data.

You should not expect our method to work on just any random combination of input and output datasets (e.g.

catskeyboards

). From our experiments, we find it works better if two datasets share similar visual content. For example,

landscape paintinglandscape photographs

works much better than

portrait painting landscape photographs

.

zebrashorses

achieves compelling results while

catsdogs

completely fails. See the following section for more discussion.

Failure cases

Our model does not work well when the test image is rather different from the images on which the model is trained, as is the case in the figure to the left (we trained on horses and zebras without riders, but test here one a horse with a rider). See additional typical failure cases here. On translation tasks that involve color and texture changes, like many of those reported above, the method often succeeds. We have also explored tasks that require geometric changes, with little success. For example, on the task of

dogcat

transfiguration, the learned translation degenerates into making minimal changes to the input. We also observe a lingering gap between the results achievable with paired training data and those achieved by our unpaired method. In some cases, this gap may be very hard -- or even impossible,-- to close: for example, our method sometimes permutes the labels for tree and building in the output of the cityscapes photos->labels task.

Citation

If you use this code for your research, please cite our paper:

@inproceedings{CycleGAN2017, title={Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkss}, author={Zhu, Jun-Yan and Park, Taesung and Isola, Phillip and Efros, Alexei A}, booktitle={Computer Vision (ICCV), 2017 IEEE International Conference on}, year={2017} }

Related Projects:

pix2pix-Torch | pix2pixHD |BicycleGAN | vid2vid | SPADE/GauGAN
iGAN | GAN Dissection | GAN Paint

Cat Paper Collection

If you love cats, and love reading cool graphics, vision, and ML papers, please check out the Cat Paper Collection.

Acknowledgments

Code borrows from pix2pix and DCGAN. The data loader is modified from DCGAN and Context-Encoder. The generative network is adopted from neural-style with Instance Normalization.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.