Need help with neural-doodle?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

alexjc
9.5K Stars 866 Forks GNU Affero General Public License v3.0 122 Commits 50 Opened issues

Description

Turn your two-bit doodles into fine artworks with deep neural networks, generate seamless textures from photos, transfer style from one image to another, perform example-based upscaling, but wait... there's more! (An implementation of Semantic Style Transfer.)

Services available

!
?

Need anything else?

Contributors list

Neural Doodle

.. image:: docs/Workflow.gif

Use a deep neural network to borrow the skills of real artists and turn your two-bit doodles into masterpieces! This project is an implementation of

Semantic Style Transfer 
_ (Champandard, 2016), based on the
Neural Patches 
_ algorithm (Li, 2016). Read more about the motivation in this
in-depth article 
_ and watch this
workflow video 
_ for inspiration.

The

doodle.py
script generates a new image by using one, two, three or four images as inputs depending what you're trying to do: the original style and its annotation, and a target content image (optional) with its annotation (a.k.a. your doodle). The algorithm extracts annotated patches from the style image, and incrementally transfers them over to the target image based on how closely they match.

NOTE: Making a

#NeuralDoodle
is a skill. The parameters in the script were adjusted to work well by default and with the examples below. For new images, you may need to adjust values and modify on your input data too. It takes practice, but you can reach almost photo-realistic results if you iterate! (
Ask for advice here or see examples 
_.)
  1. Examples & Usage 
    _
  2. Installation 
    _
  3. Troubleshooting 
    _
  4. Frequent Questions 
    _

IMPORTANT: This project is possible thanks to the

nucl.ai Conference 
_ on Creative AI, July 18-20. Join us in Vienna!

|Python Version| |License Type| |Project Stars|


.. image:: docs/Landscape_example.png

1. Examples & Usage

The main script is called

doodle.py
, which you can run with Python 3.4+ (see setup below). The
--device
argument that lets you specify which GPU or CPU to use. For the samples above, here are the performance results:
  • GPU Rendering — Assuming you have CUDA setup and enough on-board RAM, the process should complete in 3 to 8 minutes, even with twice the iteration count.
  • CPU Rendering — This will take hours and hours, even up to 12h on older hardware. To match quality it'd take twice the time. Do multiple runs in parallel!

The default is to use

cpu
, if you have NVIDIA card setup with CUDA already try
gpu0
. On the CPU, you can also set environment variable to
OMP_NUM_THREADS=4
, but we've found the speed improvements to be minimal.

1.a) Image Analogy

The algorithm is built for style transfer, but can also generate image analogies that we call a

#NeuralDoodle
; use the hashtag if you post your images! Example files are included in the
#/samples/
folder. Execute with these commands:

.. code:: bash

# Synthesize a coastline as if painted by Monet. This uses "*_sem.png" masks for both images.
python3 doodle.py --style samples/Monet.jpg --output samples/Coastline.png \
                  --device=cpu --iterations=40

Generate a scene around a lake in the style of a Renoir painting.

python3 doodle.py --style samples/Renoir.jpg --output samples/Landscape.png
--device=gpu0 --iterations=80

Notice the Renoir results look a little better than the Monet. Some rotational variations of the source image could improve the quality of the arch outline in particular.

1.b) Style Transfer

If you want to transfer the style given a source style with annotations, and a target content image with annotations, you can use the following command lines. In all cases, the semantic map is loaded and used if it's found under the

*_sem.png
filename that matches the input file.

.. code:: bash

# Synthesize a portrait of Seth Johnson like a Gogh portrait. This uses "*_sem.png" masks for both images.
python3 doodle.py --style samples/Gogh.jpg --content samples/Seth.png \
                  --output SethAsGogh.png --device=cpu --phases=4 --iterations=40

Generate what a photo of Vincent van Gogh would look like, using Seth's portrait as reference.

python3 doodle.py --style samples/Seth.jpg --content samples/Gogh.png
--output GoghAsSeth.png --device=gpu0 --phases=4 --iterations=80

To perform regular style transfer without semantic annotations, simply delete or rename the files with the semantic maps. The photo is originally by

Seth Johnson 
, and the concept for this style transfer by
Kyle McDonald 
.

.. image:: docs/Portraits_example.jpg

1.c) Texture Synthesis

For synthesizing bitmap textures, you only need an input style without anotations and without target output. In this case, you simply specify one input style image and the output file as follows:

.. code:: bash

# First synthesis uses a darker noise pattern as seed.
python3 doodle.py --style samples/Wall.jpg --output Wall.png\
                  --seed=noise --seed-range=0:128 --iterations=50 --phases=3

Second synthesis uses a lighter noise pattern as seed.

python3 doodle.py --style samples/Wall.jpg --output Wall.png
--seed=noise --seed-range=192:255 --iterations=50 --phases=3

You can also control the output resolution using

--output-size=512x512
parameter—which also depends on the memory you have available. By default the size will be the same as the style image.

.. image:: docs/Textures_example.jpg

1.d) Script Parameters

You can configure the algorithm using the following parameters. Type

python3 doodle.py --help
for the full list of options, or see the source code.
  • --style-weight=50.0
    — Weight of style relative to content.
  • --style-layers=3_1,4_1
    — The layers to match style patches.
  • --semantic-weight=1.0
    — Global weight of semantics vs. features.
  • --smoothness=1.0
    — Weight of image smoothing scheme.
  • --seed=noise
    — Seed image path, "noise" or "content".
  • --print-every=10
    — How often to log statistics to stdout.
  • --save-every=10
    — How frequently to save PNG into
    frames
    .

2. Installation & Setup

.. image:: docs/DockerConsole.gif

2.a) Using Docker Image (recommended)

The easiest way to get up-and-running is to

install Docker 
. Then, you should be able to downloand and run the pre-built image using the
docker
command line tool. Find out more about the
alexjc/neural-doodle
image on its
Docker Hub 
page.

The easiest way to run the script from the docker image is to setup an easy access command called

doodle
. This will automatically:
  • Mount the
    frames
    folder from current directory into the instance for visualization.
  • Expose the
    samples
    folder from the current directory so the script can access files!

This is how you can do it in your terminal console on OSX or Linux:

.. code:: bash

# Setup the alias. Put this in your .bash_rc or .zshrc file so it's available at startup.
alias doodle="docker run -v $(pwd)/samples:/nd/samples -v $(pwd)/frames:/nd/frames \
                         -it alexjc/neural-doodle"

Now run any of the examples above using this alias, without the .py extension.

doodle --help

If you want to run on your NVIDIA GPU, you can instead use the image

alexjc/neural-doodle:gpu
which comes with CUDA and CUDNN pre-installed in the image. See the scripts in
docker/*.sh
for how to setup your host machine. (advanced)

2.b) Manual Installation (optional)

This project requires Python 3.4+ and you'll also need

numpy
and
scipy
(numerical computing libraries) as well as
python3-dev
installed system-wide. If you want more detailed instructions, follow these:
  1. Linux Installation of Lasagne 
    _ (intermediate)
  2. Mac OSX Installation of Lasagne 
    _ (advanced)
  3. Windows Installation of Lasagne 
    _ (expert)

Afterward fetching the repository, you can run the following commands from your terminal to setup a local environment:

.. code:: bash

# Create a local environment for Python 3.x to install dependencies here.
python3 -m venv pyvenv --system-site-packages

If you're using bash, make this the active version of Python.

source pyvenv/bin/activate

Setup the required dependencies simply using the PIP module.

python3 -m pip install --ignore-installed -r requirements.txt

After this, you should have

scikit-image
,
theano
and
lasagne
installed in your virtual environment. You'll also need to download this
pre-trained neural network 
_ (VGG19, 80Mb) and put it in the same folder as the script to run. Once you're done you can just delete the
#/pyvenv/
folder.

.. image:: docs/Coastline_example.png

3. Troubleshooting Problems

It's running out of GPU Ram, throwing
MemoryError
. Help!

You'll need a good NVIDIA card with CUDA to run this software on GPU, ideally 2Gb / 4Gb or better still, 8Gb to 12Gb for larger resolutions. The code does work on CPU by default, so use that as fallback since you likely have more system RAM!

To improve memory consumption, you can also install NVIDIA's

cudnn
library version 3.0 or 4.0. This allows convolutional neural networks to run faster and save space in GPU RAM.

FIX: Use

--device=cpu
to use main system memory.

How much GPU is being used? It doesn't seem very fast...

First make sure CUDA is installed correctly and environment variables are set, then reinstall

theano
. If everything is setup correctly, the GPU should be used regularly as the gradient calculations are offloaded. If you run NVIDIA's monitoring tool it looks something like this:

.. code:: bash # gpu pwr temp sm mem enc dec mclk pclk # Idx W C % % % % MHz MHz 0 88 63 50 25 0 0 3004 1252 0 60 63 0 1 0 0 3004 1252 0 75 63 19 9 0 0 3004 1252 0 59 63 0 1 0 0 3004 1240 0 62 63 16 3 0 0 3004 1240 0 63 64 2 1 0 0 3004 1252 0 66 63 26 4 0 0 3004 1252

The third column is the utilitazition of compute resources, and the fourth column is the use of memory. If memory is under-used you can increase resolution! If compute is under allocated too you can try running multiple scripts in parallel!

FIX: Run

nvidia-smi dmon
and check the
sm
column.

Can't install or Unable to find pgen, not compiling formal grammar.

There's a Python extension compiler called Cython, and it's missing or inproperly installed. Try getting it directly from the system package manager rather than PIP.

FIX:

sudo apt-get install cython3

NotImplementedError: AbstractConv2d theano optimization failed.

This happens when you're running without a GPU, and the CPU libraries were not found (e.g.

libblas
). The neural network expressions cannot be evaluated by Theano and it's raising an exception.

FIX:

sudo apt-get install libblas-dev libopenblas-dev

TypeError: maxpool2d() got an unexpected keyword argument 'mode'

You need to install Lasagne and Theano directly from the versions specified in

requirements.txt
, rather than from the PIP versions. These alternatives are older and don't have the required features.

FIX:

python3 -m pip install -r requirements.txt

ValueError: unknown locale: UTF-8

It seems your terminal is misconfigured and not compatible with the way Python treats locales. You may need to change this in your

.bash_rc
or other startup script. Alternatively, this command will fix it once for this shell instance.

FIX:

export LC_ALL=en_US.UTF-8

ERROR: The optimization diverged and NaNs were encountered.

It's possible there's a platform bug in the underlying libraries or compiler, which has been reported on MacOS El Capitan. It's not clear how to fix it, but you can try to disable optimizations to prevent the bug. (See

Issue #8 
_.)

FIX: Use

--safe-mode
flag to disable optimizations.

4. Frequent Questions

Q: When will this be possible in realtime? I want it as filter!

Related algorithms have shown this is possible in realtime—if you're willing to accept slightly lower quality:

  • Texture Networks: Feed-forward Synthesis of Textures and Stylized Images 
    _
  • Perceptual Losses for Real-Time Style Transfer and Super-Resolution 
    _
  • Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks 
    _

This project is not designed for real-time use, the focus is on quality. The code in this repository is ideal for training realtime capable networks.

Q: Is there an application for this? I want to download it!

There are many online services that provide basic style transfer with neural networks. We run

@DeepForger 
_, a Twitter & Facebook bot with web interface, that can take your requests too. It takes time to make forgeries, so there's a queue... be patient!

|Python Version| |License Type| |Project Stars|

.. |Python Version| image:: http://aigamedev.github.io/scikit-neuralnetwork/badge_python.svg :target: https://www.python.org/

.. |License Type| image:: https://img.shields.io/badge/license-AGPL-blue.svg :target: https://github.com/alexjc/neural-doodle/blob/master/LICENSE

.. |Project Stars| image:: https://img.shields.io/github/stars/alexjc/neural-doodle.svg?style=flat :target: https://github.com/alexjc/neural-doodle/stargazers

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.