Hand bone tracking ability to create gesture passwords to control virtual worlds

As an augmented reality technology, AR brings a deep integration of the virtual digital world and the real world. This fusion of virtual and real can not only be applied to visual interaction scenes such as virtual car display and virtual interior design, but also control the virtual world through action interaction. scene, to achieve borderless human-computer interaction.

For example, when people are shooting short videos, they can control the switching of special effects only by making specific gestures without touching the screen; they can also control the shutter shooting through gesture recognition when taking pictures; they can also try nail styles, rings, etc. The above AR application scenarios require accurate hand recognition, which requires the use of AR Engine 's hand skeleton tracking capabilities .

The hand skeleton tracking capability can identify and track the position and posture of 21 hand joint points, and output advanced hand features such as finger endpoints, palm orientation, and hand bones to form a hand skeleton model. When there are multiple hands in the image, only the recognition result and coordinate information of the hand with the clearest and highest confidence will be fed back. Currently, only designated Huawei models that can obtain depth information are supported.

Through the recognition ability of hand bones , virtual objects can be superimposed to more precise hand positions, such as fingertips, palms, etc.; using hand bones, the virtual hand can be driven to make richer and finer manipulation actions, which can Provide enhanced interactive functions for AR applications to facilitate the expansion of more novel gameplay.

hand skeleton definition

Application scenarios

1. Simple sign language translation

Hand skeleton tracking capability enables simple sign language comprehension. On the basis of extracting the key points of the bones of the hand, the hand posture is predicted according to the changes of the key points, and the set of finger states such as bending and vertical can be mapped to a set of predefined gestures. For example, the index finger is stretched straight out to represent the counting gesture "1", gestures derived through such algorithms are the basis for sign language understanding and can be applied to sign language recognition and translation.

2. Build a non-contact operation interface

In sci-fi movies, there are often scenes where characters operate the computer interface through the air, and such functions can also be realized based on the ability of hand skeleton tracking.

Through the recognition and tracking of facial bones by the camera, key node information such as fingertips can be accurately located, thereby realizing physical interaction with virtual objects. Use your fingers to press virtual buttons or other UI elements; pinch virtual objects and release menu items; stretch physical controls like small targets on larger objects; even resize objects by hand or store them in virtual pockets, etc. .

This function can be applied to mechanical control in the medical industry, locomotive interface control, etc.

3. Short video live broadcast interaction

The hand skeleton tracking capability of AR Engine can also add some hand special effects in the process of short video dissemination. Recognize user gestures and superimpose corresponding special effects or stickers to realize gesture-triggered special effects, hand special effects tracking and other effects, enriching the interactive experience during live video.

In addition, the hand skeleton tracking capability can also be used in film and television animation production, children's teaching, medical rehabilitation, smart home control and other fields.

With the rapid development of AR technology, human-computer interaction based on gesture recognition has gradually become a hot topic. How to realize natural gesture recognition that conforms to human habits is an important part of human-computer interaction interface, as an item of gesture recognition. Basic capabilities, hand skeleton tracking also highlights its importance in this field. Huawei AR Engine hopes that this capability can help developers stimulate broader research directions and more innovative practical applications.

Learn more details>>

Visit the official website of Huawei Developer Alliance
Get development guidance documents
Huawei Mobile Services Open Source Warehouse Address: GitHub , Gitee

Follow us to know the latest technical information of HMS Core for the first time~