Explore high-performance graphics rendering capabilities to cope with harsh video game environments

Content source: Huawei Developer Conference 2021 HMS Core 6 Graphics Technology Forum, the keynote speech "CG Kit Explores High-Performance Rendering on Mobile"

Speaker: Huawei HiSilicon Kirin GPU Team Engineer

Hello everyone, I am a GPU R&D engineer from the Huawei HiSilicon Kirin team. The topic I bring to you today is "CG Kit Exploring High-Performance Rendering on Mobile".

Huawei's graphics computing service (Computer Graphics Kit, referred to as "CG Kit") provides cutting-edge computer graphics rendering framework, plug-in SDK and GPU extension interface to help developers tap the hardware limit performance and greatly reduce the difficulty of development. Help developers create products with a better experience.

So, what solutions can the CG Kit

1. Vulkan rendering framework practice

Vulkan is the latest graphics standard API. Compared with OpenGL, it has certain advantages, and the standard of OpenGL has stopped evolving. In the future, graphics will basically evolve with Vulkan. For example, ray tracing on the mobile terminal is described in Vulkan and is not available in OpenGL. But the problem is that Vulkan is very flexible. Compared with OpenGL, it has a certain threshold to use. Therefore, a high-performance rendering framework based on Vulkan is needed to provide practical architecture and solutions to give full play to hardware performance.

Two, high-performance rendering plug-in

CG Kit provides a series of plug-ins with rendering capabilities such as multi-threaded rendering, camera super-dividing, volume cloud, occlusion culling, etc., to facilitate the smooth progress of the three-party cooperation. For example, last year's NetEase "Tianyu" mobile game integrated the camera super-scoring algorithm .

Third, the developer tool chain

CG Kit also provides a complete developer tool chain, we can help developers quickly locate and solve rendering function and performance issues. You can imagine that if the future tool can automatically identify some function/performance issues in the game scene and feed it back to the developer, this will greatly improve our development efficiency.

Fourth, the exploration of cutting-edge graphics technology

The CG Kit team has been exploring cutting-edge graphics technology for a long time. For example, how do we apply AI technology to graphics rendering. At present, we have been involved in AI super-segmentation, automatic 3D face modeling, and AI collaborative animation generation. The research results.

Vulkan rendering framework practice

Vulkan rendering framework practice, to provide developers with Vulkan rendering solution optimization, rendering enhancement plug-ins and rendering technical documents , for example, the document will introduce some API usage strategies, resource cache usage optimization, Vulkan compatibility issues and so on.
As mentioned earlier, Vulkan is a relatively new API, and its Command Buffer mechanism can naturally support multi-threaded rendering. Since Vulkan is more flexible, CG Kit will simplify the package for Vulkan's API, making it more convenient to use.

Secondly, CG Kit also supports physics-based rendering. PBR can truly reflect the properties of the surface lighting of the object, and the objects we render through PBR will be more realistic.

At the same time, CG Kit provides a resource manager, which can reuse a large number of pipeline resources as much as possible, thereby reducing overhead and improving rendering performance.

Finally, CG Kit also supports HDR10 display capabilities, and with the display optimization of the HDR screen, the final image will have a better dynamic range.

Static super component

Let's introduce the plug-in for the game to take pictures. The game has social attributes, and sharing is a very important part of social interaction. In the course of the game, there are often some wonderful moments that are worth sharing, such as completing a milestone task, obtaining a limited edition exquisite skin, or having completed an avatar image customization...At these moments, game players I am very eager to share these wonderful moments with my friends, so I need to use the function of taking pictures, but how to make the effect of taking pictures better?

For photo effects, CG Kit provides two solutions.

The first solution is for high-end mobile phones with independent NPU, such as phones corresponding to Kirin 980, 990 and Kirin 9000 chips. These mobile phones with independent high-end NPU can directly call the AI super-division interface, so that the screen details can be significantly increased. The style is more clear and natural. As can be seen from the comparison chart below, the details of the faces of the characters in the processed pictures are obviously improved.

For those mobile phones that do not have a separate NPU chip, we also provide a general solution called FilterSR, which is an algorithm based on a general-purpose GPU. After applying this algorithm, mobile phones without NPU can also greatly improve the picture. The effect, and compared with the traditional method, its jagged feeling and picture clarity are also greatly improved. It is worth mentioning that "Tianyu" integrates our algorithm for taking pictures.

Volume Cloud Component

CG Kit also provides a high-performance volumetric cloud plug-in for mobile terminals. Volumetric cloud technology can dynamically render highly realistic global illumination scenes in the sea of clouds in real time, and supports players to shuttle freely in the clouds. The screen can truly restore the physics of the clouds. characteristic. For example, the sense of particles rushing toward the player's face during the shuttle, as well as the real lighting effects, including the details of the edges of the clouds, will be very well presented. As can be seen from the figure below, the color of this cloud is gradually changing over time, because we have adapted the light for 24 hours.

In terms of the form of volumetric clouds, we support both real and cartoon styles, and also support the conversion of Mash of any shape into volumetric clouds, which can meet the different customization needs of developers.
From a performance point of view, on the Kirin 980 platform, we can render 1 frame within 4 milliseconds. The plug-in is displayed, and both PC and mobile plug-ins are provided. Generally speaking, developers edit and debug on the mobile terminal. , And integrated in the way of SO on the mobile terminal, and its package body size is only about 400K.

Volume fog assembly

Let’s introduce the volumetric fog components. In some games of specific game styles, the highly realistic light and fog effects will bring you a very good experience. For example, there was a very popular game last year called "Cyberpunk 2077". ", there are a lot of fog effects in it.

In the host 3A masterpiece, volume fog is generally achieved through volume rendering. Unlike surface rendering, which only needs to render the surface of the object, volume rendering needs to render the interior of the object. The rendering workload may be dozens of times or hundreds of times that of surface rendering. The hardware performance requirements are very high, which is why it has not been implemented on the mobile terminal.

The CG Kit volume fog component has made many optimizations to traditional volume rendering. I will briefly introduce two optimizations.

The first type of optimization is optimized for multi-light source scenarios. When there are multiple light sources, the workload of rendering is increased by several tens of times. In response to this problem, we have introduced light source culling technology to allow lighting calculations to occur only in the local area affected by the light source, which can greatly reduce the rendering work quantity.

The second optimization is that in order to improve the rendering efficiency, we have adopted the method of reducing the sampling rate, but after reducing the sampling rate, there will be a problem of insufficient picture detail. In order to solve this problem, we have introduced a time-domain filtering technology to reduce the current frame Blending with the result of the previous frame will also get a very good result. The data structure is also relatively adapted to these optimizations.

After the previous series of optimizations, we can perform real-time high-performance volume fog rendering on the mobile terminal. From a performance point of view, on the Kirin 980 platform, 4 milliseconds one frame rendering can be achieved; in the integrated form, it can be rendered at the same time. Provide PC terminal and mobile terminal plug-in, support PC terminal to edit and debug, mobile terminal is integrated in SO mode, mobile terminal plug-in package body is less than 500KB.

AI pinch face

Finally, I will introduce the AI pinch face. In the game scene, if it can achieve a thousand faces of an avatar, and its avatar can have the facial features of a part of the game player, it will enhance the user's sense of substitution. The AI pinch face is input in the form of a frontal selfie. After calling our AI interface, a set of pinch parameters that highly restore the user's facial features will be generated, and a 3D model will be automatically generated based on the pinch parameters. At the same time, we are pursuing geometrically similar At the same time, the real texture and the model texture are integrated, so that its texture can extract some of the player's facial features, which is more reducible. We provide a very simple interface for integration, which allows developers to adapt easily and greatly enhance their development effects.

Because we are pure end-to-side operation, the AI pinch time is less than 4s in the mid-to-high-end models, and the stability is very high. At the same time, our face-squeezing range not only supports facial features, hair styles, glasses, etc.; we also incorporate highly restored real textures, which have better restoration, and provide aesthetic adjustment functions to control the final output image. Aesthetics. In terms of development efficiency, developers only need two steps to call our functions. The first step is to freely adapt the model on the PC side. The second step is to call our interface on the mobile side by taking the user’s selfie as input. The process is very simple and convenient.

I hope you can have in-depth cooperation, thank you!

Learn more >>

Visit Huawei Developer Alliance official website
Obtain the development guide document
Huawei mobile service open source warehouse address: GitHub , Gitee

and learn about the latest technical information of HMS Core for the first time~