Use ordinary cameras to easily achieve precise motion capture capabilities

In recent years, there have been more and more practical application scenarios for motion capture capabilities, such as the production of special effects in sci-fi movies and the production of virtual game characters. Most of these use traditional motion capture solutions.

Traditional motion capture usually has two solutions: optical capture and inertial capture .

Optical motion capture needs to stick reflective signs on key points of the human body, and track the cursor position on the human body from different angles through multiple infrared cameras. It has the advantages of high accuracy and low delay. The disadvantage is that there are strict requirements on the venue and the equipment is cumbersome to wear , The price is also very expensive. The inertial motion capture system uses inertial sensors such as gyroscopes and accelerometers to capture the rotation information of the key bones of the human body. Compared with optical motion capture, it is convenient to wear, but has lower accuracy and is susceptible to geomagnetic interference.

Whether it is an optical or inertial motion capture system, it is necessary to wear professional motion capture equipment, and the cost is high. And Huawei's 3D modeling service (3D Modeling Kit) newly launched motion capture capabilities, only need RGB camera to capture the rotation information of key points of the human body, so that everyone can achieve motion capture through mobile phones.

Technical Support

So, apart from auxiliary equipment, what technologies did Huawei use to achieve low-threshold, high-precision motion capture capabilities?

Human detection technology

You need to locate the position of the human body from the image before performing motion capture. The human body detection technology can return to the bounding box containing the human body, and cutting out the corresponding area according to the bounding box can remove the redundant background, which is convenient for subsequent pose estimation.

3D human pose estimation technology

The most critical part of the motion capture ability is that the monocular pose estimation algorithm based on deep learning can estimate the 3D position of the key points of the human body and the rotation information of the bones from the RGB image, which can be applied to a wide range of actual scenes.

Model acceleration and compression technology

The motion capture algorithm will eventually be deployed on the end side. Due to the limitations of storage space and power consumption, the delay and size requirements of the model are very strict. On the one hand, it is necessary to design a suitable lightweight network structure for the CPU of the mobile device. On the other hand, parameters of the trained float32 model are quantized, converted into an int8 model, and optimized for some operators, which reduces the size of the model while significantly improving the inference speed.

Application scenario

The most extensive and most direct application of motion capture capabilities is to drive virtual images, and there are many needs in virtual live broadcasts, games, and movie production. For example, in a game, using motion capture capabilities, users can upload a video to capture the actions of characters in it, and generate corresponding data to drive the characters in the game.

In addition, in the field of short video, motion capture capabilities are also promising. Model-based motion capture algorithms can output human body meshes, and various special effects can be added on this basis.

In addition, the motion capture capability has more application scenarios to be developed and unlimited use value. Huawei's motion capture capability has greatly reduced the barriers to development and use in various industries.

Learn more >>

Visit Huawei Developer Alliance official website
Obtain development guide document
Huawei Mobile Services open source warehouse address: GitHub , Gitee

and learn about the latest technical information of HMS Core for the first time~