DeepFaceDrawing: Generate face images from sketches

In this article, the idea of using a face design or sketch to make a photo of a face is presented. Applications of the technology include character design, education and training, facial morphing, and suspect portraits.

If you are interested, you can watch the video introduction first:

problem to be solved

The idea of this post is to design an app to use this face sketch to draw an image of a real face, which has various applications in police, movie shooting and face painting. One of the main criteria of this idea is that it is simple and straightforward, very easy to use, even if you are not a good painter, you can use this app to create real faces.

There have been solutions to this problem in the past, using deep learning in some cases, but most of the time they take a fixed drawing as input and try to make it more complete by adding details and generate realistic faces.

From left to right: real image - HED - APDrawingGAN

From left to right: Canny Edges - Photocopy Filter in Photoshop - Simplification of Photocopy Filter in PS

How is the problem solved? (Architecture and Ideas)

data set

A new dataset was created using face images from the CelebAMask-HQ dataset and used to train and test the model.

Only face-to-face images are collected in this database without any other additions such as glasses, masks, etc.

The edge detection process adopts an overall nested edge detection method, and the next step is to use the Photocopy Filter in Photoshop to improve the quality of the design and remove image noise. The database includes 17,000 pairs of real images and their designs, including 6,247 pairs of male images and 11,456 pairs of female images. However, this dataset is not very large and new data is needed to improve the algorithm. In this idea, the ratio of training set to test set is 1 to 20.

Architecture

The network structure of the system consists of three main modules: Component Embedding (CE), Feature Mapping (FM) and Image Synthesis (IS).

1. The CE (Component Embedding) module is an autoencoder with 5 features or main components that define the face, namely left eye, right eye, nose and mouth. (mouth) learns through other components.
2. The FM (Feature Mapping) and IS (Image Synthesis) modules together form a deep learning subnet to place and combine facial components to create a realistic face. where FM is the feature decoder encoded in the CE module, which converts the feature vector into a 32-channel feature map instead of a 1-channel prototype, which improves the performance of the model.

This mod uses the idea of shadow-guided, which is used to help those who are less proficient in drawing and painting. In the background there will be a blurred image of the face design which the user can use to complete the design and draw a better design with more detail.

The input to this network is 512 × 512. In the first stage, these images enter the first sub-network or CE to learn each encoder associated with each face component separately, and in the second stage, each face component uses K in its specific class according to the similarity Nearest neighbor algorithm and is placed into the graph. In the next step of FM, each component is plugged into its decoder and real components for production. The new components are then combined to create a truly new face. This step is performed in SI using a conditional GAN architecture technique, which guarantees that all face components are in their original positions.

An example is as follows:

Other features

In this model, the amount of attention and accuracy of each feature of the face can be changed, and the degree of fidelity of each feature can be adjusted, for example, the nose is more like the mouth and the mouth is less like. Another feature is that each eye can be individually designed to ensure that the eyes may not necessarily be the same.

Compared

Summarize

In this idea, we try to draw a face and separate its main components (e.g. left and right eyes, nose, mouth) from the rest and teach a deep learning network to find their equivalents in real images and transform them for the real look. There are many challenges in this direction, DeepFaceDrawing tries to solve it with new ideas and previous knowledge, but there are still problems with this idea, which we hope will be solved in the next version, and the quality and accuracy of this application will improve.

The official website, paper and DEMO code are all here:

https://www.overfit.cn/post/cf48c77252ad43928cea5c581b90a88e

Author: Ehsan Mrh