3D modeling service provides more efficient and professional 3D production capabilities, "building" developers

3D modeling service (3D Modeling Kit) is another technology opening of HMS Core in the field of graphics and images. The positioning of 3D modeling products is to provide fast, simple, and low-cost 3D production capabilities, and to gradually open them to developers who have the ability to produce 3D models and animation games. Developers do not need special hardware (RGBD\LiDAR, etc.) to use the common view information matching between captured images to estimate the depth information and complete the acquisition.

3D modeling service has opened up the ability of object modeling, as well as the three core capabilities of plane material generation and motion capture. Next, it will open up further capabilities for human modeling and facial capture. The following is a brief introduction to the developers.

1. 3D object modeling ability

(1) Introduction to 3D object modeling capabilities

3D object modeling capabilities of Different from the traditional way that modelers use 3D software to make models, or the way to model through scanners, our positioning is to do fast, low-cost UDC production on the mobile side, only need ordinary RGB cameras, through Shoot multiple images of objects at different angles and upload them to the cloud to automatically generate 3D geometric models and textures of objects. In terms of hardware requirements, we currently support Android, iOS, and WEB APR, forming a full-terminal support form.

for supported categories. Because this is a photography method, the texture of the object needs to be as rich as possible, non-reflective, medium in size, and rigid. Typical objects include: commodity plush toys, bags, shoes, etc., furniture such as sofa, Pillow, cultural relics can be bronze, stone, wood and so on.

The size of the supported objects is between 10 cubic centimeters and 2 cubic meters. Of course, larger physical sizes can be modeled, but the modeling time will be longer. If the modeled image is at 1080P resolution, the entire modeling time is about 5 minutes, and the final output format is also the standard 3D model format, including obj and gltf formats. The last important point is that the SDK package supporting this capability is only 83K in size, which is very friendly to our APP developers.

(2) Suggestions on 3D object modeling and acquisition methods

If the developer wants to build a relatively complete model, there are still certain requirements for the acquisition conditions.

first, environment . It is recommended to place a single acquisition object on a solid-colored surface.

second, light . The lighting needs to be soft and avoid too dim lighting or strong shadows on one side.

Third, shoot . To focus, cover a uniform and sufficient multi-view angle, including upward, head-up, and top-down angles, a total of more than 50 shots, and the camera moves as slowly as possible.

Fourth, acquisition process . Adjacent photos should have at least 70%-80% overlap, and the display should be as large and complete as possible. It is required to ensure that the shooting is free from defocus, motion blur, and jitter blur.

Finally, object can be established by flipping the continuation shot to achieve the rendering of the complete model of the object .

(3) Application scenarios

For object modeling, a typical scenario is e-commerce. The following picture shows the modeling of shoes and cultural relics that we did. We placed a small turntable, and then placed a small light box, and then the modeling can be realized according to the above method. Compared with 2D product display, 3D display can bring users a better experience. In the e-commerce industry, 3D product display can provide a differentiated experience, and improve user conversion through simulation methods such as "seeing, trying, wearing, and wearing". , In the cultural and expo industry, cultural relics can be applied to digital preservation and multimedia display of cultural relics after 3D digital modeling.

Second, the generation of plane materials

(1) 3D material definition

The second capability is the generation of flat materials. Appearance in graphics, also known as material, is a property that describes how light interacts on the surface and interior of an object. The surface appearance is defined by a material model and a set of control parameters. As can be seen from the following group of pictures, the picture on the left, if you don’t look carefully, the feeling of the whole scene is very weak, but if you add the walls, floors, oil drums, table tops, and these materials, the realism is immediately displayed. .

In the process of material production, what pain points do developers and users have?

First, textures collected from the real world usually need to be processed by professional software for art before they can be used in rendering to achieve realistic effects. For example, the processing of removing the influence of lighting, adjusting the convex and concave, scale, etc., the process is time-consuming, laborious and difficult to standardize; Second, the traditional texture map processing effect relies heavily on art experience and specific renderers, and texture materials are reused between different projects and renderers difficult, less accumulated project, production costs remain high .

To address these pain points, we offer two ideas:

first one uses the deep learning inference network to generate texture elements that meet the PBR standard with one click, improving the efficiency and quality of texture production; The second one, solidifies the experience and production specifications of technical art into data that follows the PBR standard , reuse materials between different projects and renderers through data, and inherit the experience of art production.

(2) Introduction to material generation capabilities

The material generation capability provides the ability to convert RGB images into PBR materials. It only takes one or more RGB images to generate five material maps with one click.

To use this capability, only ordinary RGB mobile phones are required, no ToF or LiDAR is required, and all Android models are supported; the supported material types mainly include concrete, marble, rock, gravel, brick, plaster, clay, metal, wood, bark, Leather, fabric, lacquer, plastic, synthetic materials and other materials; support diffuse map, normal map, specular map, roughness map, height map texture types; for the resolution of the input image, it can currently support the output of 1k~8k textures, The input image needs to be free of seams, bright spots, shadows, and reflections; for the final production effect, the evaluation index of SSIM is about greater than 0.9, and the effect is very good.

(3) Material library

Developers can use a variety of material libraries according to their needs, filter scenes and types according to their needs, and provide PBR material maps with 1k-8k resolution. At the same time, it supports functions such as query, preview and download.

3. Motion Capture Capability

The just-released motion capture capability is to input continuous video frames or RGB images through an ordinary monocular camera to accurately and quickly estimate the three-dimensional information of 24 key points of human skeleton. Developer integration only needs to use ordinary mobile phones, input video streams or images, and can estimate human skeletons, which can support common movements such as standing, walking, dancing, squatting, sideways, and bending over.

The input format of the video stream and RGB image, it is recommended that the resolution of the picture in the video is not less than 320 320 px. Too low resolution will affect the detection accuracy. At the same time, the resolution is not higher than 1920 1080 px, too high resolution will affect the detection frame rate. The output format is also more important. The motion capture capability can output the quaternion and 3D coordinates of 24 human skeleton key points at the same time. In some engines, it can directly drive the matching.

for more details >>

Visit Huawei Developer Alliance official website
Get development guidance document
Huawei Mobile Services open source warehouse address: GitHub , Gitee

us and know the latest technical information of HMS Core for the first time~