Implement SLIC algorithm to generate pixel art

Preface

Pixel wind first appeared in 8bit video games. Due to the size of computer memory and the single display color, only a small number of pixels can be used to present content, but many classic pixel games have been achieved. With the improvement of memory capacity and screen resolution, the limitation of memory and display media is no longer a problem, and the pixel style has gradually evolved into a unique creative style.

The general drawing process of pixel drawing includes hooking, coloring, etc., and drawing pixel by pixel requires a lot of time. Some popular art methods, such as line drawing and painting, have gradually appeared automated or semi-automatic generation methods. This article will implement the SLIC [1] algorithm from scratch and implement a tool for generating pixel art.

What is the SLIC algorithm

The reason why pixel painting is not easy to draw is because direct down-sampling cannot accurately capture key pixels, and it easily leads to loss of edge information, and the generated pixel painting is often unsatisfactory. Manual hooking and coloring are all for selecting appropriate pixels. Therefore, our problem becomes how to select the appropriate pixels for coloring.

First, introduce a concept-super pixel. Super pixel is an image segmentation technology proposed and developed by Xiaofeng Ren in 2003. It refers to an irregular pixel block with certain visual significance composed of adjacent pixels with similar texture, color, brightness and other characteristics [1].

By dividing the picture into super pixels, similar pixel clusters can be obtained. Similar pixels are filled with the same color, and the resulting pixel picture will be more reasonable.

The methods of super pixel segmentation include contour extraction, clustering, and gradient ascent. SLIC super pixel segmentation algorithm (simple linear iterative clustering, simple linear iterative clustering ) proposed by the paper [1] is one of them. It is based on the K-means clustering algorithm, clustering according to the color and distance characteristics of the pixels to achieve good segmentation results, and several SLIC super pixel segmentation algorithm, 060d570f3b2d9c has the advantages of simplicity, flexibility, good effect, and fast processing speed.

How to implement the SLIC algorithm

The basic process of SLIC

1. Image preprocessing.

   The image RGB color space to the CIE-Lab color space. The Lab color space is more in line with human visual perception of colors. The distance in this space can reflect the color difference that people feel, and the related calculations are more accurate.

   Lab color space also has three channels, namely l , a , b , wherein l representative luminance, values ranging [0,100] , a represents from green to red component values ranging [-128,127] , b represents the blue to yellow component, The value range is [-128,127] .

   RGB is no direct conversion formula between 060d570f3b3039 and LAB RGB to XYZ color space and then to LAB . For the code, see the complete code at the end of the article.

2. Initialize the cluster centers.

   Determine the number of super pixels according to the parameters, that is, how many regions need to be divided into. Assuming that the picture has N pixels, it is expected to be divided into K super pixels, each super pixel size is N/K , the distance between adjacent centers is S=Sqr(N/K) , and K cluster coordinates are obtained.

   

3. Optimize the initial cluster centers. 3*3 neighborhood of the cluster center, the pixel with the smallest gradient is selected as the new cluster center.

   Regarding the image as a two-dimensional discrete function, the gradient is the derivation of this function. When the value of adjacent pixels changes, there will be a gradient, and the gradient of the pixel on the edge is the largest. Moving the cluster center to the place with the smallest gradient can prevent it from falling onto the edge contour and affect the clustering effect.

   The gradient calculation of the discrete gradient is not deduced in detail here. Since it contains a number of squares and roots, the calculation is relatively large, and it is generally simplified to the operation of approximating the square and the square root with the absolute value. The simplified formula for the gradient of the pixel with the calculated coordinate of (i,j)

   

   Among them, （i+1,j） and （i,j+1） are the coordinates of the point on the right side of the pixel and the point below the pixel. l(a,b) is the luminance channel value l （a，b） coordinate.

4. Calculate the distance between the pixel point and the cluster center.

   Calculate the distance between the pixel point and each cluster center in the neighborhood 2S*2S in the area of the cluster center distance S.

   

   Here the distance used is Euclidean distance, total distance D the dc color distance ds spatial distance between the two part composition. The formula is as follows:

   

   If directly l , a , b , x , y spliced into a vector calculating the distance, when the change in the size of the super-pixel, x , y value may be taken to be very large, if such a map 1000*1000 , spatial distance can reach 1000*Sqr(2) , And the maximum color distance is only 10*Sqr(2) . As a result, the weight of the ds

   Therefore, it needs to be normalized, divided by the maximum value, which is the initial width of the super pixel, S , and map the value to [0,1] .

   And it will give a color space distance to a fixed value m adjusted affecting the right color and the spatial distance from the weight, m the range [1,40] .

   The distance formula becomes

   

   When m is larger, the value of the color space divided by m is smaller, that is, the larger the weight of the spatial distance, the generated pixels will be more regular in shape. When m smaller, the color distance weight is larger, and the super pixel will be more at the edge. It is compact, and the shape and size are relatively irregular.

5. Pixel classification.

   Mark the category of each pixel as the category with the smallest distance from the cluster center.

6. Recalculate the cluster centers.

   Calculate the average vector value of all pixels belonging to the same cluster, and get the cluster center again.

7. Iterate the process of 4～6

   Until the distance between the old cluster center and the new cluster center is less than a certain threshold or reaches a certain number of iterations, generally speaking, when the number of iterations reaches 10 , the algorithm can reach convergence.

8. Cluster optimization.

   At the end of the iteration, there may be isolated pixels that do not belong to the same connected domain as the cluster center, and the connected algorithm can be used to assign them to the nearest cluster label.

   The paper did not give a specific implementation algorithm. However, the application scenario of this article is to generate pixel paintings, which will downsample the pixels, and will not refine to each pixel. Therefore, this article does not perform clustering optimization processing.

   

To sum up, the SLIC algorithm process is roughly the same as K-means , and iteratively calculates the cluster with the smallest distance. The difference is that only the S distance of the cluster center are calculated, which reduces a lot of calculation.

Generate pixel art

Based on the SLIC algorithm, we can already divide a picture into N super pixels. The pixels in each superpixel are similar. In other words, each pixel is classified as a super pixel and has a cluster center. Then assign the color of the pixel to the color of its cluster center to get the effect we want.

Set a certain step size stride , use Canvas , and assign the pixel to the color of the cluster center stride

And everyone's subjective feelings about pixel art are inconsistent, in order to let users have more choices and get their own satisfactory results. More manual intervention parameters can be exposed, such as canceling the termination condition of clustering optimization, and changing the number of iterations and the final step size of the pixel value to be set by the user. Manually set parameters include

• Super pixel size blocksize ; the blocksize super pixel segmentation.
• The number of iterations is iters ; the iters , the more accurate the segmentation result and the longer the calculation time.
• The color space weight weight ; the weight greater the influence of the color on the segmentation result.
• Take the pixel step size stride ; the stride , the closer the generated pixel image is to the super pixel, the more delicate it is.

Implement user interaction interface

As a tool, a user interaction interface is naturally needed. The front-end interface is based on HTML/Javascript/CSS , and Canvas API used to draw image content, while the user interaction panel selects the dat.gui [3] library. dat.gui is a lightweight graphical interface library, very suitable for parameter modification, often used as a demonstration of visual Demo. Supported parameter types include Number , String , Boolean , custom functions, etc. Corresponding response events can be bound to different properties, and the event will be triggered automatically when the property value changes.

Add the following properties and events for generating pixelated tools:

• When the iters、stride、blockSize、weight (color space weight m) parameter changes, re- SLIC algorithm and re-draw the calculation result;
• Add Upload image and Export image buttons to support users to upload pictures and download pixelated pictures;

Superimpose a layer of Canvas Canvas canvas layer of the drawn image to visualize the results of the algorithm and add the following functions

• grid switch controls whether to draw a pixel grid;
• Centers switch controls whether to display cluster centers;
• Contours switch controls whether to display cluster edge contours;

The drawing of the cluster center point Centers ctx.fillRect center point coordinates.

To draw the super pixel contour Contours , the contour points need to be calculated first.

You can compare each pixel with the surrounding 8 pixels. If the number of pixels with different cluster centers is greater than 2 , it means that there are more than two different types of points around the pixel, and this point is a contour . The effect is as follows:

Finally, you have a simple tool for generating pixel art.

Experience address

full version code address (JS version)

references

[1] Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Su ̈sstrunk S. SLIC superpixels. Technical Report. IVRG CVLAB; 2010.

[2] Gerstner T , Decarlo D , Alexa M , et al. Pixelated image abstraction with integrated user constraints[J]. Computers & graphics, 2013.

[3] https://github.com/dataarts/dat.gui

Welcome to pay attention to the Bump Lab blog: aotu.io

Or follow the AOTULabs official account (AOTULabs) and push articles from time to time.