2022 Taobao Maker Festival 3D Live Virtual Camp Technology Highlights Revealed

On August 24, 2022, the Taobao Maker Festival will be unveiled! In order to explore the next-generation Internet immersive shopping experience, in addition to creating a 20,000-square-meter panoramic “future camp” in the offline Canton Fair exhibition hall, this Maker Festival also launched the first online virtual interactive space “3D Crazy”, where 10,000 people can be online at the same time. City - Festival of Creation Virtual Camp". In the 3D virtual camp, users can experience the high-definition game texture in a lightweight way, and join the world of "3D people and goods yard" in the form of virtual characters, interact with players, interact with 3D commodities, and meet the love of the new generation of consumer groups The appeal of play and love to create brings a new consumption experience with innovative technology.

Let's take a look at the experience & technical highlights of the 3D live virtual camp:

Lightweight naked-eye 3D "real-life game"

Once you liked a 3D game, the client downloaded several gigabytes, but had to uninstall it when you wanted to play a new game, it took up too much memory! In the virtual camp of the Maker Festival, you quickly loaded a naked-eye 3D, interactive, live broadcast, and even buy-to-buy high-definition picture "game", and your Taobao App "doesn't get bigger at all", the mobile phone loads The speed is "whoosh"~

Technology Highlights: The Challenge of Cloud Costs

The traditional development of 3D game applications on the mobile terminal often requires the integration of dozens or hundreds of M game engines in the terminal test, as well as the need to download several gigabytes of material resources, which is currently unrealistic in the Taobao app. The cloud rendering method can solve this problem very well. With the help of the powerful GPU in the cloud to complete high-definition and complex scene rendering, users do not need to download large resource files, nor do they need to install any content. The only thing they need is to play real-time rendered media. flow.

But the biggest problem here is the cost of cloud rendering machines. In order to reduce the cost of cloud rendering machines, in addition to cloud rendering, we have implemented performance optimization strategies such as model surface reduction, dp merging, texture optimization, and reducing video material resolution. In terms of machine scheduling, a dynamic expansion and contraction strategy in time periods is implemented in combination with business scenarios to maximize machine utilization.

3D people and 3D goods meet for the first time in a 3D virtual world

The 3D users who once dressed up in "Taobao Life" "walked" out of their small homes for the first time and came to the live 3D virtual camp of the Creation Festival. Here, you can dance freely, light a bonfire, view 3D products, watch sellers’ live broadcasts, and interact with players. 3D people, 3D goods, and 3D environments are combined to give you an immersive and wonderful camp adventure experience.

Technical Highlights: Construction and Interaction of People and Goods Yards

Human-to-human interaction. The 3D characters in Taobao Life are placed in the 3D virtual world, and they first need a certain amount of asset conversion and rendering costs. The user runs around the camp, which involves real-time synchronization of the user's location. For real-time user roaming, we use frame synchronization technology to update user location information at a fixed frequency. Ideally, users will not feel stuck, but it is inevitable that network congestion will cause inter-frame jitter, and two adjacent locations If the synchronization time exceeds a certain interval, it will cause jumping. In order to make up for this deficiency, we use a motion compensation algorithm when rendering characters. Simply put, the algorithm simulates the transition of the user's position to make the transition smoother. At the same time, in terms of synchronization of interactive data, the AOI grid algorithm is adopted to solve the problem of data synchronization of multiple people on the same screen. In this way, the 3D characters running freely in the virtual world will be smoother and smoother.

Human-Commodity Interaction. It is not easy for users to freely view the details of 3D products in a virtual scene. We have two options: one is mobile rendering and the other is cloud rendering. There are also relatively mature solutions for mobile rendering in Taobao. Unfortunately, the rendering engine of the mobile terminal and the rendering engine of the cloud are not the same set, so the same product model cannot be used universally. At the same time, dynamic download of the model, surface reduction optimization, etc. must be considered. The problem is, in order to avoid these disadvantages, we still choose the cloud rendering solution, by dynamically switching the camera perspective, and responding to the interactive instructions of the user in real time, so as to achieve the experience of rotating and zooming to watch 3D products.

1080P resolution cool HD quality

Once you tried a small game on the page in the mobile app, it was like a mosaic, running and jumping and dropping frames. The new 3D virtual camp of the Creation Festival, with high-definition picture quality and smooth effect, the stage lighting can dynamically track you as a "star"~

Technical Highlights: Content Production and Real-time Transmission for Cloud Rendering

In the cloud, we use Unreal Engine to render real-time images, and combine technologies such as dynamic mirror movement, dynamic stage lighting, multi-perspective characters, and particle effects to make the images look more cool and enhance user interaction. In order to transmit high-definition pictures to users' mobile phones, we need to comprehensively consider the balance of picture quality, freeze, and delay. With the help of the GRTN transmission network jointly built by Taobao and Alibaba Cloud, as well as the self-developed codec algorithm, the maximum To ensure the clarity of the picture quality.

Interactive virtual screen does not freeze

Once you were in the live broadcast room of Taobao anchors and counted down "3, 2, 1, start grabbing!" You felt the coolness of ultra-low latency spikes. But off-screen you cannot directly intervene in the actions of the host in the live broadcast room through interaction. In the 3D virtual camp, you can say hello while dancing, and you can click on the 3D products of the merchants to watch the explanations of your favorite anchors. You can really "communicate with the merchants and the anchors in the same frame".

Technical Highlights: Full Link Ultra-Low Latency

Different from non-interfering image transmission, to achieve a smooth experience with ultra-low latency in an interactive environment, we need to complete the upload of user instructions, the rendering of real-time images within 100 milliseconds, and return the rendered images to the user's mobile phone. First of all, it is necessary to comprehensively consider the user's machine, network, and access point to dynamically select the streaming node closest to the user, so as to adjust the streaming strategy of the cloud (including the choice of clarity and encoding method). Secondly, in the transmission process, it is also necessary to adopt weak network control strategies such as real-time bandwidth feedback, FEC, and dynamic buffers to combat network jitter, so that the entire link delay can reach a reasonable position, and finally realize the interaction in the virtual world. Ensure a smooth interactive experience.

Epilogue

The 3D virtual camp of the Maker Festival is the first online attempt of Taobao's virtual interactive space "3D Crazy City" series. In the next step, we will combine cloud real-time rendering and XR/CG technology to realize virtualized scene customization, support real-time user interaction on a larger scale, and bring consumers a more immersive interactive experience.