RealX

Gadgets world

Create your own «neural paintings» with DeepStyle & Ubuntu

Create your own

Neural networks can do a lot. They can interpret images, understand our voices and translate conversations But did you know that they can also paint?

A recent research paper (entitled «Artistic Style Neural Algorithm») has sparked a wave of online discussions with several striking illustrative examples. In essence, the article discusses a technique for training a deep neural network to separate the art style from the structure of an image and combine the style of one image with the structure of another. The result of all this is that you can train a huge neural network to turn photographs into «neural paintings» that look like they were painted by famous artists — so-called «digital fakes».

Here are some examples from the article. The first image is the original. The later images are generated results and the drawing from which the sample was taken is shown in thumbnail.

paper images

The original researchers did not release their code, unfortunately. However, some intrepid programmers have been copying their results from the last few days and their code is available open source on the Internet. All you need to get started is a Linux machine and a little patience.

Today I will tell you how to do it and show you some of my own results. You can think of it as a loose extension of our DeepDream tutorial. It’s a little tricky, but anyone with a Linux machine can follow along — no programming experience required.

Software setup

First, if you’re not in a hurry or don’t have a Linux computer, you can still play around with DeepStyle with DeepForger bot on Twitter (send him the image and style and he will eventually respond with the results you want). If you want to process more images quickly (and with more control over the results), read on for a tutorial.

First of all, make sure you have the latest version ubuntu (14.04 is what I used). You should have at least a few extra gigabytes of hard drive space. For more information, see our tutorial on dual booting Ubuntu alongside Windows. You will also need superuser rights, so make sure you do this before you continue.

Basic tools

From the very beginning, this is an open source project, so we want to install git . Git is the gold standard for version control software . Pretty much every open source project worth knowing about is hosted on Github.

To download and install Git, just open a terminal and type » sudo apt-get install git » and agree to the requirements of the installer.

Next: we’re going install some basic tools, necessary for the operation of the software.

Install Lua first. This is the language in which the tool is written. It’s pretty simple. Just type » sudo apt-get install lua5.2 ” and follow the installation process.

Second, we’re going to get Luarox. It’s a tool that makes it easy to install other tools (don’t you love Linux?). To do this, enter » sudo apt-get install luarocks » and follow sudo apt-get install luarocks by installation.

Third, we are going to install Luajit. This is a timely compiler for Lua that will make our lives a little easier. Just type » sudo apt-get install luajit «.

So far, so good.

Frameworks

Next, we’re going to install Torch, a scientific computing and machine learning framework that forms the backbone of the application. Unfortunately, it cannot be installed using apt-get (the standard Ubuntu package manager).

Luckily, they have a one-line installer that uses command line magic. Go back to your terminal and type » curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-all | bash curl -s https://raw.githubusercontent.com/torch/ezinstall/master/install-all | bash «.

When you’re done, type » luajit -ltorch «. This will bring up the cutter interface and check that everything is installed correctly.

Way out of this.

Now we are going to install loadcaffe — package for the neural network. Install its only dependency by typing » sudo apt-get install libprotobuf-dev protobuf-compiler «. You can then install the package itself using » sudo luarocks install loadcaffe" .

Double checking dependencies

Finally, we’re going to pre-update some things, just to make sure everything runs smoothly.

Enter » sudo luarocks install image to make sure your image pack is up to date. Then type » luarocks install nn «, which will do the same for your «nn» package.

Setting the Deep Style

Good! At this point, we are ready to actually install the software itself. For the sake of cleanliness, create a new folder in your home directory (‘mkdir DeepStyle’). Then enter it using » cd Deepstyle «. Now type » sudo git clone https://github.com/jcjohnson/neural-style.git".

Next, we need to download the model. Make a cup of coffee or something, it will take some time. Go back to the terminal and type » sudo sh models/download_models.sh «. This will start a long, complicated download process. If it fails due to permission errors, try giving yourself read and write permissions to the appropriate folders using chmod.

Using Deep Style

Okay, we’re fine. Using the software is quite simple.

Make sure you are in DeepStyle/neural-style directory in the terminal. Now you need some images to work with. Download them from the internet (or whatever), then copy them to DeepStyle/neural-style folder using the file browser.

Now you can use the command line to process individual images. The format is pretty simple:

th neural_style.lua -style_image YOURPAINTINGHERE.jpg - content_image YOURPHOTOHERE.jpg -gpu -1

(Obviously you will need to replace chunks in ALL CAPS with your filenames.)

This will start the neural network. It will run for about an hour, spitting out new partially converging images every few minutes until it runs out. Flag -gpu -1 stops attempts to access your GPU.

After hours of trying (and bricking the operating system multiple times) I couldn’t get Ubuntu and CUDA to play well with my GPU (NVIDIA GTX 970). With more luck, you’ll want to install CUDA and cudann.torch (see the github repo for more info). If not, then it’s okay — it will still run on your processor; it will be a little slower.

If you have any problems getting all this work just ask me in the comments and I will do my best to help you.

Results

Here are a few images I’ve generated over the past couple of days. The results are mixed, but many of them are impressive.

Zack-paden-composite

Click here for full resolution.

This is my friend Zach on a trip to Yellowstone. The style comes from an abstract painting created by Teresa Paden. I was curious to see how the system would use an image without any structure. The results are pretty neat and you can definitely see the resemblance to the style of the image.

Jersey City-Demuth-Composite

Click here for full resolution.

This book is courtesy of one of my favorite artists, Charles Demuth (See New Church Incense and Figure 5 on Gold). Interestingly, Demuth is one of the main visual inspirations for Team Fortress 2’s art. as you can see from the style image.

I gave him a picture of the city of Jersey that I found on Wikimedia. The results… are pretty good. He didn’t capture the angularity of Demuth’s style, but he certainly picked up the soft, textured look and color palette.

flowers-OKeefe-composite

Click here for full resolution.

This is an attempt at a synthetic O’Keeffe using a fairly mundane picture of some of the colors I found. The results are, frankly, impressive. Aesthetically, this is one of my favorite results. The richness of O’Keeffe’s colors and shapes is clearly visible. The layered edges of the flower petals become the edges of the leaves in the background. The flowers themselves dissolve into flowers, becoming almost abstract.

It would be a good picture if a person did it. I really want to take a couple of days to render this higher resolution version and create it.

shannon-picasso-composite

Click here for full resolution.

Here’s my friend Shannon in her Picasso Halloween costume. Interestingly, the device chose to paint the lower part of her face white (similar to the color layout of the Picasso piece). I’m not sure if it was a coincidence or not, but the results are amazing. It looks like he correctly identified Shannon’s hair on the left side and redrawn it using the color and hairline in the style image. The same goes for her hat.

This is one of the parts where the limitations of the technique start to become clear. If Picasso had really painted Shannon, he would have discarded the structure of her face and distorted her features to achieve the desired effect. This system does not understand this kind of high-level concept and is only able to mimic the superficial aspects of the style, such as dark angled lines and color palettes.

eiffel-vangogh-composite

Click here for full resolution.

Pretty simple: a photo of the Eiffel Tower and another Starry Night Van Gogh. It does an excellent job of rendering a Van Gogh-style cloud, despite the lack of clouds in the original image. It also does a good job of translating the scene from day to night.

I wasn’t sure why he decided to make the tip of the Eiffel Tower a pillar of fire. It looks cool, but it’s not entirely justified from the input. Then I realized that there are thirteen long vertical yellow stripes in the image of the style in the form of reflections in the water. This is quite a massive cluster given so little training data. The poor thing has probably learned that any high-contrast vertical edge must be one of those reflections. You can see more extraneous vertical streaks faintly in the clouds.

nebula-vangogh-composite

Click here for full resolution.

The same Van Gogh painting, but this time I gave it real stars. In this case, part of the pillars of the Eagle Nebula. I like the results — though, once again you can see his obsession with the yellow stripes. Each vertical part of the post becomes a bright, wobbly yellow line. It is also clearly upset by the greens that were not in the training data and is doing its best to get rid of them in favor of blues and blacks.

Technology

Some of the results of this are extremely convincing, although the technique has clear limitations. Some of the images have lousy composition, and the system struggles with more abstract artists like Picasso — who were known to love to distort their subject, dissipating its features. The algorithm captures its angled lines and clashing colors, but is still subject to the image’s pixel values. It lacks the understanding that you need to deviate too far from the source material.

What worries me about all this is that I don’t consider these limitations to be fundamental.

The approach used here — training a network on one image and using it to build another — is basically a kind of hack. This gives the network very little data to work with. A more advanced version of this application will use the web, which has information about many paintings and maybe even real images, to give it enough context about the image it’s trying to «draw».

A deep understanding of style can only exist in a larger context. You cannot extract it from a single image. Developing an architecture that gives the system access to wider data can allow it to gain a more «human» view of an image and how artists represent different elements of the real world. Such a network can produce images that are more abstract and have better composition. Such algorithms would stop being a cool toy (like this one) and become a way to create real, original art.

It’s a very peculiar idea in a way.

Create your own images

If you get a disappointing result, you can experiment a bit with the options to get more convincing results. The full list is on Github. The important ones are

Once you get everything you like, please leave your most interesting images in the comments. I’m really interested to see what you guys come up with.

Image Credits: Human Brain Artist via Shutterstock

Похожие записи

Car Amplifier Classes

ABS Safe Driving Tips

8 Rules for Instant Messaging at Work