Treasure Shengteng AI application recommendation! CANN takes you to experience the joy of automatic coloring of black and white images~

Abstract: belongs to the wheat field on the "you" side, the sky, the river, the horses, and the trees...what color will it be?

Black and white and color

It seems to divide the era in half~

I saw the chaos and prosperity on this end, a dazzling array of colors, and the fast-decreasing colors.

Open a black and white photo

I can see the imprint passed down through generations, a beauty that will last forever

I can see but I can’t see the scenery that "you" can see

This seemingly inadvertent glance is like a glimpse in my heart

Daunting

But I can't help quietly fantasize

The wheat fields, sky, river, water, horses and trees belonging to "you"...

What color will it be?

Freeze frame right now

I would like to give "you" thousands of colors,

Take you to experience this prosperous scene together~

The principle of black and white image coloring model

This application based on the Ascend AI heterogeneous computing architecture CANN , and is built using a black and white image coloring model. It is suitable for Atlas 200 DK and Atlas 300I and other related reasoning products.

Use convolution to extract features for objects (including background color) (the grayscale image represented by the L channel), and then use convolution to classify them, so as to try to give the color expectation of each pixel of the grayscale image (ab channel) . Then, the value of the ab channel is resized to the width and height of the original image and superimposed with the L channel, and then converted to an RGB image to obtain a color image. The input of the model is the L channel data (224, 224, 1), and the output is the ab channel data (56, 56, 2).

We can also understand the input and output of the original model through the test script of the open source model, as well as the calculation process before input (also called preprocessing) and the processing process after output (also called postprocessing).

Preprocessing process: RGB format is read into Lab, resize to 224224, L channel is extracted, and the mean is subtracted (-50)
Post-processing process: the result of the inference ab channel, resize to 224224, merge with the input L into Lab, convert to RGB, and save as a jpeg image

Original link of the model: https://github.com/richzhang/colorization

Prepare the model

Ascend Tensor Compiler ATC (Ascend Tensor Compiler) can convert open source framework network models (such as Caffe, TensorFlow, etc.) and single operator description files defined by Ascend IR into offline models supported by Ascend AI processors, and model conversion In the process, the optimization of operator scheduling, weight data rearrangement, memory usage optimization, etc. can be realized, and the preprocessing of the model can be completed without the equipment.

Use ATC to convert the black and white image coloring model to the offline model supported by the Ascend AI processor. The commands are as follows:

atc --input_shape="data_l:1,1,224,224" --weight="./colorization.caffemodel" --input_format=NCHW --output="./colorization" --soc_version=Ascend310 --framework=0 --model="./colorization.prototxt"

Application development

This application uses the modular design shown in the figure below to complete the inference output of a picture through the coordination of various modules; the reasoning for the video is the same, you can directly refer to the source code in the open source warehouse. I won't repeat it here.

Wherein each module main function points follows:

Operation management resource application

Used to initialize the internal resources of the system and a fixed calling process.

Load model files and build output memory

To load offline model data from a file, the user needs to manage the memory of the model running. According to the model loaded in the memory, the basic information of the model includes the data buffer size of the model input and output data; the model output memory is constructed from the basic information of the model, which is Prepare for the next model reasoning.

Data preprocessing

Receive the image data passed by the outside world, preprocess the read image data, and then construct the input data of the model.

Model Reasoning

Perform model reasoning based on the input data of the constructed model.

Analytical reasoning results

According to the model output, analyze the inference result of the model. Use opencv to save the converted color image data into a picture file.

Preprocessed image

In the image preprocessing part, use the imread interface of opencv to read the picture, and the read out is in BGR format.

The model input is 224×224, so the read image needs to be resized to 224×224. Normalize the data after resizing.

As stated in the algorithm design part, the model uses the Lab color space, so it is necessary to convert the BGR format to Lab format data. This model uses the L channel data to predict the possible ab space data, so the L channel data must be separated from the Lab data. Then subtract the average value from the obtained data to get the input data required by the model.

Model reasoning

In the model reasoning part, the key process is explained as follows:

Call the aclmdlCreateDesc interface to create a data type that describes the basic information of the model;
modelDesc_ = aclmdlCreateDesc();

Call the aclmdlGetDesc interface to get the basic information of the model according to the model ID returned during model loading. aclError ret = aclmdlGetDesc(modelDesc_, modelId_);
Prepare input and output data for model inference;
Perform model inference, call the aclmdlExecute interface; aclError ret = aclmdlExecute(modelId_, input_, output_);
Obtain the result of model inference for subsequent processing.
Free up memory. Call the aclrtFree interface to release the memory on the Device. Release the data of the relevant data type. After the model inference is over, it is necessary to call the aclDestroyDataBuffer interface and the aclmdlDestroyDataset interface in time to release the data that describes the model input, and first call the aclDestroyDataBuffer interface, and then call the aclmdlDestroyDataset interface. If there are multiple inputs and outputs, the aclDestroyDataBuffer interface needs to be called multiple times.

Post-processing part

For image post-processing, opencv is used. For model reasoning, the predicted ab space data is obtained.

First, resize the obtained data back to the original image size, and then merge it with the original image L channel data to obtain a complete Lab image.

Convert the Lab image back to the BGR format to save it as a jpeg image and get the colored image.

Coloring black and white images is an art and a technique~

AI "Black and White Image Coloring" application has been launched, you can color with just one click!

This case address: https://www.hiascend.com/zh/developer/mindx-sdk/cartoon/990674866img?fromPage=1

In this era, a new color can represent a new emotion and a new perspective.

Monotonous black and white seems to be insufficient to meet our needs~

Let AI technology take you to interpret the different beauty~

At the same time, click on the online experiment, you can also experience how the application was developed with CANN;

The source code of this application is open source in both gitee and github. Click on github and gitee above to go directly to the source code warehouse, and you can perform secondary development based on the source code.