Keynote Speech at Huawei Developer Conference: 3D Modeling Service Makes Content Efficient Production

Content source: Huawei Developer Conference 2021 HMS Core 6 Graphics Technology Forum, the keynote speech "3D Modeling Service Enables Efficient Production of 3D Content".

Speaker: Huawei Consumer Cloud Service AI Algorithm Expert

3D modeling service (3D Modeling Kit) is a new Kit released by Huawei at the beginning of July this year. Our positioning of 3D modeling products is to provide fast, concise, and low-cost 3D production capabilities, and open them to 3D models. , Animation production and other ability demanding developers.

For 3D modeling services, we have opened up the ability to model objects, as well as the ability to generate planar materials and motion capture. Next, we will open up further capabilities for human body modeling and facial capture. There are also many application scenarios for 3D modeling, which can be applied to e-commerce models, 3D printing, teaching models, game development and animation production.

3D object modeling capabilities

Introduction to 3D object modeling capabilities

First look at the ability of 3D object modeling, which is different from the traditional way of modelers using 3D software to make models, or the way of modeling through scanners. Our positioning is to do fast and low-cost UDC on the mobile terminal. The production of, does not need the depth information of the picture, only an ordinary RGB camera, by taking multiple images of the object from different angles, and uploading to the cloud, the automatic generation of the 3D geometric model and texture of the object can be realized.

In terms of hardware requirements, we currently support Android and iOS, and we may open WEB APR next, which will form a full-terminal support form.

For the supported categories, because this is the method of photography, it is required that the texture of this object needs to be as rich as possible, non-reflective, medium-sized, and rigid. Typical objects include: commodity plush toys, bags, shoes, etc., furniture Types such as sofas and pillows, cultural relics can be bronze, stone, wood, etc.

The supported object size is approximately between 10 cubic centimeters and 2 cubic meters. Of course, larger physical sizes can also be modeled, but the modeling time will be longer. If the modeled picture is at a resolution of 1080P, the entire modeling time is approximately less than 5 minutes, and the final output format is also the standard 3D model format, including the obj and gltf formats. The last more important point is that the SDK package supporting this capability is only 83K in size, which is very friendly to APP.

3D object modeling and acquisition method recommendations

If you want to build a relatively complete model, there are still certain requirements for the acquisition conditions.

First, the environment. It is recommended to place a single collection object on a solid-colored plane.

Second, light. The lighting needs to be soft, and avoid too dark lights or strong shadows on one side.

Third, shooting. To focus, cover a uniform and sufficient multi-view angle, including upward, horizontal, and top view angles totaling more than 20 shots (50 or more is recommended), and the camera moves as slowly as possible.

Fourth, during the collection process, adjacent photos have at least 70%-80% overlap, and the display is as large and complete as possible, and it is required to ensure that the shooting is free of virtual focus, motion blur, and jitter.

Lastly, the bottom of the object can be established by flipping and shooting to achieve a complete model of the object.

Application scenarios

For object modeling, the typical scenario is e-commerce. The picture below is about the modeling of shoes and cultural relics we did. We bought a small turntable, put a small light box, and then modeled in the above way. Compared with 2D product display, 3D display can bring a better experience to users. In the e-commerce industry, 3D product display can provide a differentiated experience, and improve user conversion through simulation methods such as "seeing, trying, putting on, and wearing" , In the cultural and cultural industry, cultural relics can be used for digital preservation and multimedia display of cultural relics after three-dimensional digital modeling.

Plane material generation

3D material definition

The second ability is the generation of flat materials. Appearance in graphics, also known as material, is a property that describes how light interacts on the surface and inside of an object. The surface appearance is defined by a material model and a set of control parameters. As can be seen from the next set of pictures, the picture on the left, if you don’t look carefully, will have a very weak impression of the whole scene, but if you add these materials to the wall, the floor, the oil drum, and the tabletop, the sense of reality is immediately displayed. .

What are the pain points for developers and users during the material production process?

First, textures collected from the real world usually need to be processed by professional art software before they can be used in rendering to achieve realistic effects. For example, the process of removing the influence of light, adjusting the convexity and the scale, the process is time-consuming, laborious and difficult to standardize; second, the traditional texture mapping processing effect is heavily dependent on art experience and specific renderers, and texture materials are difficult to reuse between different projects and renderers. , The project accumulation is small, and the production cost remains high.

In order to solve these pain points, we provide two ideas:

first type is to use deep learning inference network to generate texture elements that meet the PBR standard with one click, and improve the efficiency and quality of texture production; Reuse materials between different projects and renderers, inheriting art production experience .

Introduction to material generation capabilities

The material generation capability provides the ability to convert RGB images into PBR materials. You only need to take one or more RGB images to generate four texture maps with one click.

To use this ability, only ordinary RGB mobile phones are required, ToF or LiDAR is not required, and all Android models are supported; the supported material types mainly include concrete, marble, rock, gravel, brick, plaster, clay, metal, wood, bark, Leather, fabric, lacquer, plastic, synthetic materials and other materials; support diffuse map, normal map, specular map, roughness map and other texture types; for the resolution of the input image, it currently supports 1k~4k texture output and input image There are no seams, no bright spots, shadows, and reflections in the middle; for the final production effect, it is estimated by the evaluation index of SSIM, which is about greater than 0.9, and the effect is very good.

You can see how to quickly create a house through the following demonstration. First, you need to quickly generate a texture map through the material generation ability, and then drag the texture map onto the ball. When using it, copy the texture ball to the white film. Realized. After copying, you can get the final real effect after offline or online rendering. This is the ability of material generation.

Motion capture ability

Finally, the motion capture capability we just released. Motion capture is to input continuous video frames or RGB images through a normal monocular camera to accurately and quickly estimate the three-dimensional information of the key points of 24 human bones.

Motion capture ability also only needs to use an ordinary mobile phone, input video stream or image, you can estimate the human skeleton, which can support about 7 to 8 common actions such as standing, walking, and running.

The input format of video stream and RGB image is recommended to have a resolution of 320~1080P. If the resolution is larger, it may increase the time, but the effect is not significantly improved, so you can do an image scaling first to produce a Better results.

The output format is also more important. Our motion capture capability can simultaneously output the quaternion and 3D coordinates of 24 key points of human bones. In some engines, it can directly drive matching.

Regarding latency, our CPU can reach 30 frames FPS, while NPU can reach 80 frames FPS.

We provide two integration methods, one is the basic package method, if you are a Huawei phone, you only need to integrate the basic package 200k. In actual application, you can download the algorithm package, for example, on Xiaomi mobile phones or other mobile phones, you can integrate the FullSDK mode.

The above APP that integrates object modeling capabilities is called 3D Live Scanner.

Before accessing 3D modeling services, this APP could only rely on devices with in-depth information, which made it unavailable for some users and lacked the ability to model individual objects. After integrating the 3D modeling service, it not only supports RGB mobile phones, but also supports background removal, and the accuracy of the model effect is greatly improved. According to the data feedback from CP, after integrating this capability, in Europe this region, its download volume has increased by about 190% within one month of this year, and the overall revenue has also increased significantly. In addition, after using the APP, customers reported that the modeling effects produced by Huawei's 3D modeling capabilities have exceeded the professional modeling capabilities of the PC version that everyone often uses.

The above is my sharing, thank you!

more details>>

Visit Huawei Developer Alliance official website
Obtain the development guide document
Huawei mobile service open source warehouse address: GitHub , Gitee

Follow us and learn about the latest technical information of HMS Core for the first time~