The main methods and landing practice of video image color enhancement

background

As part of the image quality enhancement technology in video post-processing, color enhancement refers to adjusting the saturation characteristics of pictures and video images to make the colors of the images richer and more realistic, and to improve the subjective visual experience of people. Due to the diversity of device cameras, some cameras have problems such as insufficient saturation, gray images, and dull colors during video capture. These situations are especially prone to occur when shooting outdoor scenes and scenes with rich colors. These issues can be ameliorated by post-processing tweaks to the algorithm. Color enhancement combined with video noise reduction, dark light enhancement, contrast adjustment, edge enhancement and other technologies can greatly improve the video quality.

■Schematic diagram of color enhancement effect (left: original image, right: after enhancement)
Original image source: HEVC standard test stream

Design of Color Enhancement Algorithm

Post-processing generally has two different goals for color adjustment, "restore" and "good-looking". "Restoration" means that the picture can truly present the true color of the scene without color cast. "Good-looking" means that the color level of the picture is rich enough and natural, which can improve the visual comfort of people, make the sky bluer and greener, and the picture produced is more in line with people's preferences, which is a relatively subjective indicator. Color enhancement is primarily concerned with the goal of "looking good." At present, there is a lack of objective evaluation standards and public data sets for color enhancement. According to some collected information, the design goals of the following color enhancement algorithms can be roughly summarized:

1. After adjustment, it can improve people's subjective feeling of picture quality. Whether this goal is achieved mainly depends on the subjective judgment and scoring of developers, reviewers and users.

2. The algorithm is adaptable. Adaptation means that for different images, or different parts of the same image, the adjustment intensity varies according to the image characteristics. Avoid over-saturation of already vibrant colors.

3. No obvious flaws after adjustment. The more common problems with color enhancement are oversaturation, loss of picture details, color casts, or defects such as color layering. In addition, it is easy to enhance the color noise in the dark parts of the picture.

4. For the video frame sequence, the processing of color enhancement cannot cause sudden changes in the color of the picture. For example, for some algorithms that rely on global statistics, the sudden appearance of the light source may cause a sudden change in the picture effect, resulting in a jump in the picture effect.

5. Protect the skin color of the portrait. A simple color enhancement algorithm can easily lead to the problem that the skin color of the portrait becomes "orange". Since the skin color of a person is different from the color of an object, there is an empirically reasonable range, and adjustment beyond this range will make the portrait look unreal. Therefore, it is necessary to protect the skin to some extent.

6. It is necessary to consider whether the cooperation of color enhancement and other image quality adjustment modules in the entire video processing process will cause the overall effect to deteriorate.

The above points can be regarded as a self-check table for algorithm design. It is not necessary to strictly satisfy all of them, but if one of them has a serious problem, the usability of the algorithm will be greatly reduced.

Choice of Color Model

The color enhancement method generally converts the RGB color to a suitable color model, such as HSV, YUV, Lab color model, etc., and then applies the designed algorithm to adjust the specific channel. Generally, it is hoped to extract the components that represent the vividness of the color and adjust it separately, so that the adjustment will not affect the light and shade, and will not cause color cast.

■Adjustment flow diagram

In addition, there are currently methods based on neural networks for image enhancement, where the network adjusts the image itself without the need for a human to design formulas. The enhancement method of deep learning generally realizes the simultaneous adjustment of color, contrast and brightness enhancement in a network to achieve the effect of image quality enhancement, which is divided into black box method and white box method.

Since our engineering requirements for color enhancement are to implement fast, real-time, lightweight algorithms on the mobile side with highly controllable effects, the effects need to be decoupled and coordinated with modules such as beauty, dark light enhancement, and denoising. Therefore, non-neural network color enhancement methods are a better choice.

The first problem faced by color enhancement methods is how to choose a suitable color model for algorithm design.

Saturation adjustment based on HSV color model

The adjustment method based on HSV saturation is to convert the RGB color model to the HSV color model, where HSV represents (Hue) hue, (Saturation) saturation and (value) lightness respectively. Adjusting only the saturation can enhance the vividness of the color without affecting the shading and hue.

Common adjustment methods include overall lift, proportional increase, or curve adjustment to increase the overall saturation. The gamma curve shown in the figure below is a commonly used adjustment curve.

■ Curve adjustment diagram

However, the saturation adjustment increases the intensity of all colors at the same time, which is relatively rough and may lead to local oversaturation and the disappearance of local details.

A certain degree of oversaturation occurred in the skin color (left: original image, right: after enhancement)

■The details of the folds of the clothes are lost after adjustment (left: original image, right: after enhancement)
Original image source: HEVC standard test stream

For this problem, the concept of Vibrance is proposed. When adjusting the vibrance, it will intelligently enhance the softer (that is, less saturated) colors in the picture, while keeping the original saturated colors as they are.

based on natural saturation

The concept of natural saturation was first proposed by photoshop, and the focus is on adaptability. After adjustment of natural saturation, it is generally more natural than saturation adjustment. The following describes two algorithms for natural saturation adjustment.

The first performs statistics and adjustments directly on the RGB channels. The process of its natural saturation adjustment is:

1. Calculate the mean and maximum value of each pixel r, g, and b, denoted as rgb_avg and rgb_max respectively.

2. Calculate the k value: k = ( abs( rgb_max - rgb_avg ) / 127.0 ) * Vibrance. Among them, Vibrance v is the adjustment intensity coefficient, ranging from 0 to 100.

3. Use the same formula to adjust r, g, and b respectively. Taking R as an example, r = (rgb_max - r) * k + r.

This adjustment method can perform different adjustments for pixels with different saturation, which can better avoid the occurrence of oversaturation. However, it can be known from the formula that the adjustment tends to move the rgb value closer to the same rgb_max value, which may not ensure that the color remains stable. The adjustment strength of different colors varies greatly, and color cast will occur.

The second is adaptive adjustment through brightness and saturation. The adjustment process of its natural saturation is:

1. Calculate the luma value from RGB: luma = 0.2126 r + 0.7152 g + 0.0722 *b.

2. Calculate the saturation value from rgb: saturation = max(r , g, b) - min(r, g, b).

3. Calculate the k value: k = 1.0 + Vibrance * (1.0 - satuation / 255.0), where Vibrance is the adjustment strength 0~1.

4. Use the same formula to adjust r, g, and b respectively. Taking r as an example, r = luma (1.0 - k) + r k.

Similarly, it can be known from the formula that the adjustment tends to bring the rgb value closer to the same luma value, and the color cannot be guaranteed to remain stable, and color cast will occur.

■ Enhancement based on Vibrance (Left: Original Image Right: After Enhancement)

From the perspective of the color model, color adjustment based on color spaces such as RGB and HSV is non-uniform to human visual perception, which is likely to produce color casts, so a more suitable color model is needed. The Lab color model is a color model improved and named by CIE in 1976. It is a device-independent color model and a color model based on physiological characteristics, which is more suitable for color enhancement.

based on Lab/Lch color model

The Lab color model consists of three elements, one element is the lightness (L), and a and b are the two color channels. The Lch color model uses the same color space as Lab, but it uses c for the saturation value and h for the hue angle. It can be considered that ch is the polar representation of ab. Since the Lch model is established based on physiological characteristics, it means that if only the c-saturation value is adjusted, the effect of ensuring no color cast can be achieved.

The automatic color enhancement algorithm of the open source image processing software Gimp is adjusted using the Lch color model. The processing flow is as follows:

1. Convert RGB to LCH.

2. Traverse the image pixels to find the maximum value c_max and minimum value c_min of C.

3. Stretch the C of each pixel: c = (c - c_min) / (c_max - c_min) * 100.

4. Convert LCH back to RGB.

■Before processing

■After processing

Adjustment in Lch space is much more scientific than adjustment in HSV. From the actual effect, there is basically no color cast. However, Gimp's method is mainly aimed at pictures. When used in video, due to the existence of global statistics, finding the maximum and minimum values will increase a certain amount of calculation, and it will be greatly affected by the strongest and weakest points of the image. If the strongest point and weakest point suddenly leave or enter the screen, the effect may appear abruptly, so this method is not suitable for video processing. In actual engineering, the adjustment formula needs to be redesigned to remove the dependence on the global statistical point.

Color Noise Suppression

The noise of image acquisition is divided into color space, which can be divided into luma noise luminance noise and chroma noise color noise. In the application scenario of color enhancement, a relatively easy problem is to amplify the color noise in the picture at the same time, so it should be avoided as much as possible in the design process of the algorithm.

Noise of the same intensity will appear far more noticeable in flat areas with less saturation than in brightly colored areas. In the gray area of color images, noise often appears in the form of color noise. If color enhancement is not distinguished, these noises will be enhanced at the same time, as shown in the following figure:

■A schematic diagram of enhanced color noise (left: original image, right: enhanced)

According to the statistical characteristics of color noise, during engineering, it is necessary to judge the noise and reduce the adjustment intensity of color enhancement in these places. The effect after noise suppression is as follows:

■Noise suppression effect

skin tone protection

Since there is an empirically reasonable range for human skin color, adjusting beyond this range will make the portrait look unreal. Therefore, when the color is enhanced, it is necessary to protect the skin color of the person to avoid excessive adjustment. To protect skin tone, first identify the skin tone, and then reduce the effect of color enhancement for the part of the skin tone. Most of the skin color recognition algorithms are based on statistical features in a certain color space, including a large number of empirical values. Generally, the mainstream methods of skin color recognition are as follows:

Skin color recognition based on RGB color model

Using the RGB color model directly is a relatively simple method, because generally the incoming data is mainly RGB. The more commonly used method is the conditional judgment method. Conditional judgment is performed on the value of RGB, and it is judged as skin color when it falls within the empirical range. Among them, there are two judgment conditions: uniform light and side light. In actual operation, these two judgment conditions are generally used at the same time. As long as one of them is met, it is judged as skin color.

In addition to this, quadratic polynomial pattern detection is also available in the RGB color model. The model consists of two quadratic polynomials of the normalized RG plane and a circle equation. As long as the values of these three formulas fall within a certain empirical range, it is judged as skin color.

The RGB-based skin color recognition method is the easiest to operate, and generally does not require color model conversion. However, since the color does not separate brightness and color, the judgment formula is generally cumbersome, and the accuracy of skin color recognition is greatly affected by lighting conditions.

Skin color recognition based on YCbCr color model

Generally, the YUV we hear actually refers to YCbCr. The YCbCr color model decomposes color into luminance value Y and two-dimensional chromaticity value CbCr, which can only judge chromaticity and avoid the influence of lighting conditions. Using the YCbCr color model, the more direct skin color recognition method is the range judgment method. After converting from RGB to YCbCr, as long as the two-dimensional plane composed of CbCr falls within the empirical range, it is considered to be skin color. Different empirical ranges can be found in the literature. Generally, one of them can be used, for example, 133 <= Cr <= 173, 77 <= Cb <= 127.

In addition to directly judging the range, YCbCr can also use an ellipse-based skin color recognition method. The method is based on empirical statistical data. For a two-dimensional plane composed of CbCr, as long as it falls within the ellipse, it is considered to be skin color, and the accuracy is higher than that of pure range judgment. This skin color detection method is adopted by opencv. The ellipse parameters in opencv are: [113, 155.6] is the center of the ellipse, [23.4, 15.6] is the major and minor axes, and the tilt angle is 43.0.

■Oval skin color CbCr area schematic

Skin color recognition based on HSV color model

The HSV color model also separates brightness and color, so it also has a strong anti-interference ability for light changes.

A common method for skin color recognition based on HSV is the method based on Gaussian model. This approach is adopted by the open source processing library GPUImage. The basic idea is: convert RGB to HSV space, and calculate the distance between h and skinHue: dist = abs(h – skinHue) / 0.5, where skinHue is the statistical mean of the Gaussian model, generally 0.05 is used. The statistical parameter related to the variance of the Gaussian model is skinHueThreshold, which is generally 40. The Gaussian weight is obtained by the exp(-dist dist skinHueThreshold) formula. This method only calculates the weight (probability) of skin color recognition, and the skin color detection result can be obtained later by setting a threshold (such as >0.95).

In addition, the skin color recognition method based on HSV also has a range judgment method, which is generally judged by the empirical range of H, S, and V provided in the literature. The operation is relatively simple and will not be listed in detail here.

Schematic representation of skin color recognition results (top: original image bottom: skin color recognition results)

■Color enhancement effect under skin tone protection

Color enhancement algorithm landing effect

After adding noise suppression and skin tone protection, we got the final color enhancement algorithm effect.

■The effect of the color enhancement algorithm after adding noise suppression and skin color protection (left: original image, right: after enhancement)
Original image source: HEVC standard test stream

Summarize

In general, in the process of implementing our color enhancement algorithm, adding noise suppression and skin color protection algorithms can achieve better and more stable effects in most scenes. However, there are still many areas for improvement, such as how to better adapt to the scene and picture characteristics, how to make skin color recognition more accurate, and so on. In addition, what is discussed here is only color enhancement. To achieve better image quality enhancement, it is necessary to adjust the contrast and brightness, denoising algorithms, and even pseudo-HDR algorithms. How to output a better image by combining different algorithms As a result, it is also a topic that requires constant iterative research.

Dev for Dev column introduction

Dev for Dev (Developer for Developer) is a developer interactive innovation practice activity jointly initiated by Agora and the RTC developer community. Through various forms of technology sharing, communication and collision, and project co-construction from the perspective of engineers, the power of developers is gathered, the most valuable technical content and projects are mined and delivered, and the creativity of technology is fully released.