Turn your two-bit doodles into fine artworks with deep neural networks, generate seamless textures from photos, transfer style from one image to another, perform example-based upscaling, but wait... there's more! (An implementation of Semantic Style Transfer.)
.. image:: docs/Workflow.gif
Use a deep neural network to borrow the skills of real artists and turn your two-bit doodles into masterpieces! This project is an implementation of
Semantic Style Transfer_ (Champandard, 2016), based on the
Neural Patches_ algorithm (Li, 2016). Read more about the motivation in this
in-depth article_ and watch this
workflow video_ for inspiration.
doodle.pyscript generates a new image by using one, two, three or four images as inputs depending what you're trying to do: the original style and its annotation, and a target content image (optional) with its annotation (a.k.a. your doodle). The algorithm extracts annotated patches from the style image, and incrementally transfers them over to the target image based on how closely they match.
NOTE: Making a
#NeuralDoodleis a skill. The parameters in the script were adjusted to work well by default and with the examples below. For new images, you may need to adjust values and modify on your input data too. It takes practice, but you can reach almost photo-realistic results if you iterate! (
Ask for advice here or see examples_.)
Examples & Usage_
IMPORTANT: This project is possible thanks to the
nucl.ai Conference_ on Creative AI, July 18-20. Join us in Vienna!
|Python Version| |License Type| |Project Stars|
.. image:: docs/Landscape_example.png
The main script is called
doodle.py, which you can run with Python 3.4+ (see setup below). The
--deviceargument that lets you specify which GPU or CPU to use. For the samples above, here are the performance results:
The default is to use
cpu, if you have NVIDIA card setup with CUDA already try
gpu0. On the CPU, you can also set environment variable to
OMP_NUM_THREADS=4, but we've found the speed improvements to be minimal.
The algorithm is built for style transfer, but can also generate image analogies that we call a
#NeuralDoodle; use the hashtag if you post your images! Example files are included in the
#/samples/folder. Execute with these commands:
.. code:: bash
# Synthesize a coastline as if painted by Monet. This uses "*_sem.png" masks for both images. python3 doodle.py --style samples/Monet.jpg --output samples/Coastline.png \ --device=cpu --iterations=40
Generate a scene around a lake in the style of a Renoir painting.
python3 doodle.py --style samples/Renoir.jpg --output samples/Landscape.png
Notice the Renoir results look a little better than the Monet. Some rotational variations of the source image could improve the quality of the arch outline in particular.
If you want to transfer the style given a source style with annotations, and a target content image with annotations, you can use the following command lines. In all cases, the semantic map is loaded and used if it's found under the
*_sem.pngfilename that matches the input file.
.. code:: bash
# Synthesize a portrait of Seth Johnson like a Gogh portrait. This uses "*_sem.png" masks for both images. python3 doodle.py --style samples/Gogh.jpg --content samples/Seth.png \ --output SethAsGogh.png --device=cpu --phases=4 --iterations=40
Generate what a photo of Vincent van Gogh would look like, using Seth's portrait as reference.
python3 doodle.py --style samples/Seth.jpg --content samples/Gogh.png
--output GoghAsSeth.png --device=gpu0 --phases=4 --iterations=80
To perform regular style transfer without semantic annotations, simply delete or rename the files with the semantic maps. The photo is originally by
Seth Johnson, and the concept for this style transfer by
.. image:: docs/Portraits_example.jpg
For synthesizing bitmap textures, you only need an input style without anotations and without target output. In this case, you simply specify one input style image and the output file as follows:
.. code:: bash
# First synthesis uses a darker noise pattern as seed. python3 doodle.py --style samples/Wall.jpg --output Wall.png\ --seed=noise --seed-range=0:128 --iterations=50 --phases=3
Second synthesis uses a lighter noise pattern as seed.
python3 doodle.py --style samples/Wall.jpg --output Wall.png
--seed=noise --seed-range=192:255 --iterations=50 --phases=3
You can also control the output resolution using
--output-size=512x512parameter—which also depends on the memory you have available. By default the size will be the same as the style image.
.. image:: docs/Textures_example.jpg
You can configure the algorithm using the following parameters. Type
python3 doodle.py --helpfor the full list of options, or see the source code.
--style-weight=50.0— Weight of style relative to content.
--style-layers=3_1,4_1— The layers to match style patches.
--semantic-weight=1.0— Global weight of semantics vs. features.
--smoothness=1.0— Weight of image smoothing scheme.
--seed=noise— Seed image path, "noise" or "content".
--print-every=10— How often to log statistics to stdout.
--save-every=10— How frequently to save PNG into
.. image:: docs/DockerConsole.gif
The easiest way to get up-and-running is to
install Docker. Then, you should be able to downloand and run the pre-built image using the
dockercommand line tool. Find out more about the
alexjc/neural-doodleimage on its
The easiest way to run the script from the docker image is to setup an easy access command called
doodle. This will automatically:
framesfolder from current directory into the instance for visualization.
samplesfolder from the current directory so the script can access files!
This is how you can do it in your terminal console on OSX or Linux:
.. code:: bash
# Setup the alias. Put this in your .bash_rc or .zshrc file so it's available at startup. alias doodle="docker run -v $(pwd)/samples:/nd/samples -v $(pwd)/frames:/nd/frames \ -it alexjc/neural-doodle"
Now run any of the examples above using this alias, without the
If you want to run on your NVIDIA GPU, you can instead use the image
alexjc/neural-doodle:gpuwhich comes with CUDA and CUDNN pre-installed in the image. See the scripts in
docker/*.shfor how to setup your host machine. (advanced)
This project requires Python 3.4+ and you'll also need
scipy(numerical computing libraries) as well as
python3-devinstalled system-wide. If you want more detailed instructions, follow these:
Linux Installation of Lasagne_ (intermediate)
Mac OSX Installation of Lasagne_ (advanced)
Windows Installation of Lasagne_ (expert)
Afterward fetching the repository, you can run the following commands from your terminal to setup a local environment:
.. code:: bash
# Create a local environment for Python 3.x to install dependencies here. python3 -m venv pyvenv --system-site-packages
If you're using bash, make this the active version of Python.
Setup the required dependencies simply using the PIP module.
python3 -m pip install --ignore-installed -r requirements.txt
After this, you should have
lasagneinstalled in your virtual environment. You'll also need to download this
pre-trained neural network_ (VGG19, 80Mb) and put it in the same folder as the script to run. Once you're done you can just delete the
.. image:: docs/Coastline_example.png
You'll need a good NVIDIA card with CUDA to run this software on GPU, ideally 2Gb / 4Gb or better still, 8Gb to 12Gb for larger resolutions. The code does work on CPU by default, so use that as fallback since you likely have more system RAM!
To improve memory consumption, you can also install NVIDIA's
cudnnlibrary version 3.0 or 4.0. This allows convolutional neural networks to run faster and save space in GPU RAM.
--device=cputo use main system memory.
First make sure CUDA is installed correctly and environment variables are set, then reinstall
theano. If everything is setup correctly, the GPU should be used regularly as the gradient calculations are offloaded. If you run NVIDIA's monitoring tool it looks something like this:
.. code:: bash # gpu pwr temp sm mem enc dec mclk pclk # Idx W C % % % % MHz MHz 0 88 63 50 25 0 0 3004 1252 0 60 63 0 1 0 0 3004 1252 0 75 63 19 9 0 0 3004 1252 0 59 63 0 1 0 0 3004 1240 0 62 63 16 3 0 0 3004 1240 0 63 64 2 1 0 0 3004 1252 0 66 63 26 4 0 0 3004 1252
The third column is the utilitazition of compute resources, and the fourth column is the use of memory. If memory is under-used you can increase resolution! If compute is under allocated too you can try running multiple scripts in parallel!
nvidia-smi dmonand check the
There's a Python extension compiler called Cython, and it's missing or inproperly installed. Try getting it directly from the system package manager rather than PIP.
sudo apt-get install cython3
This happens when you're running without a GPU, and the CPU libraries were not found (e.g.
libblas). The neural network expressions cannot be evaluated by Theano and it's raising an exception.
sudo apt-get install libblas-dev libopenblas-dev
You need to install Lasagne and Theano directly from the versions specified in
requirements.txt, rather than from the PIP versions. These alternatives are older and don't have the required features.
python3 -m pip install -r requirements.txt
It seems your terminal is misconfigured and not compatible with the way Python treats locales. You may need to change this in your
.bash_rcor other startup script. Alternatively, this command will fix it once for this shell instance.
It's possible there's a platform bug in the underlying libraries or compiler, which has been reported on MacOS El Capitan. It's not clear how to fix it, but you can try to disable optimizations to prevent the bug. (See
--safe-modeflag to disable optimizations.
Related algorithms have shown this is possible in realtime—if you're willing to accept slightly lower quality:
Texture Networks: Feed-forward Synthesis of Textures and Stylized Images_
Perceptual Losses for Real-Time Style Transfer and Super-Resolution_
Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks_
This project is not designed for real-time use, the focus is on quality. The code in this repository is ideal for training realtime capable networks.
There are many online services that provide basic style transfer with neural networks. We run
@DeepForger_, a Twitter & Facebook bot with web interface, that can take your requests too. It takes time to make forgeries, so there's a queue... be patient!
|Python Version| |License Type| |Project Stars|
.. |Python Version| image:: http://aigamedev.github.io/scikit-neuralnetwork/badge_python.svg :target: https://www.python.org/
.. |License Type| image:: https://img.shields.io/badge/license-AGPL-blue.svg :target: https://github.com/alexjc/neural-doodle/blob/master/LICENSE
.. |Project Stars| image:: https://img.shields.io/github/stars/alexjc/neural-doodle.svg?style=flat :target: https://github.com/alexjc/neural-doodle/stargazers