HMS Core AR Engine 2D Image/3D Object Tracking Technology Helps Create a Smarter AR Interactive Experience

AR technology has been widely used in marketing, education, games, exhibitions and other scenarios. Through 2D image tracking technology and 3D object tracking technology, users only need to use a mobile phone to shoot, and they can achieve AR effects of flat objects such as posters and cards, as well as three-dimensional objects such as cultural relics and figures. Although 2D image tracking and 3D object tracking algorithms have made great progress in recent years, limited by the environment and other factors, the effect still has a lot of room for improvement, which is the current research and development difficulty for AR developers.

HMS Core AR Engine provides device-cloud collaborative tracking technology for 2D images and 3D objects, allowing developers to develop AR applications more simply and conveniently, allowing users to have a richer AR interaction experience.

2D Image Tracking Technology

The device-cloud collaborative 2D image real-time tracking technology has been first applied to Bilibili membership purchases. Shoppers can browse the AR special effects of products when purchasing, and interact with IP-derived products from multiple angles and at close range, so as to obtain a brand-new AR shopping experience. According to the statistics of the membership purchase background, the sales of products with AR special effects are better, and the participation rate of AR activities is about twice that of ordinary games, which helps to increase the income of the membership purchase platform.

Figure 1. Bilibili membership purchase renderings

In the traditional pure device-side 2D image tracking solution, if developers want to make new images support AR special effects, they must update the application version, which has high version maintenance costs, and with the increase of AR-capable images, the application size will also increase. growing. To this end, AR Engine adopts a device-cloud synergy architecture to solve this problem. Developers only need to upload images to the cloud to quickly release AR effects of new images. The device-side application version does not need to be updated, and the application volume will not increase accordingly.

Figure 2. Schematic diagram of device-cloud collaborative 2D image tracking technology

Device-cloud collaborative 2D image tracking technology mainly covers:

Cloud side image feature extraction module

Cloud Side Vector Retrieval Engine

End-to-side Vision Tracking Module

In terms of improving the response speed of cloud services, the device-cloud collaborative 2D image tracking technology utilizes hardware platform acceleration to develop a high-performance vector retrieval engine that cooperates with software and hardware to achieve millisecond-level retrieval response under massive features.

3D object tracking technology

In scenes such as cultural relics introduction, commodity display, AR education, etc., AR Engine also provides real-time tracking technology for 3D objects in device-cloud collaboration. This technology can realize the holographic information display effect of 3D objects

Figure 3. Rendering of real-time tracking technology of 3D objects based on device-cloud collaboration

In real life, there will be 3D objects with different texture materials, such as solid color weak texture materials such as sculptures, or reflections such as metal utensils. 3D objects will also be affected by changes in light, causing shadows on the surface. These scenes are all tracking bands for 3D objects. a big challenge. In order to realize fast and accurate identification and tracking in various scenarios, AR Engine uses a combination of multiple AI networks to complete the 3D identification and positioning function, and completes object detection, coarse positioning of pose, and pose optimization respectively.

Figure 4. Schematic diagram of device-cloud collaborative 3D object tracking technology

Device-cloud collaborative 3D object real-time tracking technology mainly covers:

Cloud-side training sample AI synthesis module

Cloud-side AI network automatic training module

Cloud-side AI network reasoning module

End-to-side Vision Tracking Module

In the algorithm training of AI network, the traditional manual labeling method of training samples often has problems such as heavy workload and long development cycle. To this end, AR Engine combines massive offline data and adversarial generation strategies to design an AI synthesis algorithm for training samples, which can avoid manual annotation, and can achieve accurate recognition of 3D objects in complex scenes only by relying on synthetic samples.

In the scene of cultural relics introduction, the cloud-based 3D object tracking technology has helped Huawei Hetu to realize a holographic cultural tour guide experience for Mogao Grottoes, helping tourists to explore the internal details of the caves at any time.

Figure 5. The rendering of the Mogao Grottoes holographic cultural tour guide by Huawei Hetu

The above technologies come from Huawei's 2012 Central Media Technology Institute, and are open to developers through AR Engine, bringing differentiated AR experiences to users in advertising marketing, education, cultural tourism and other scenarios.

For more information on AR technology, welcome to watch the 15th live broadcast of HMS Core Discovery on May 27th from 19:00-20:00, let us explore how HMS Core AR Engine helps developers build a three-dimensional world and create immersive marketing service experience.

Welcome to the official website of Huawei Developer Alliance - HMS Core AR Engine

Learn more details>>

Visit the official website of Huawei Developer Alliance
Get development guidance documents
Huawei Mobile Services Open Source Warehouse Address: GitHub , Gitee

Follow us to know the latest technical information of HMS Core for the first time~