NetEase Yunxin releases real-time interactive integration SDK for virtual images

The meta universe was caught off guard.

In just a few months, Minecraft and Roblox have entered the field of vision of more people. GREE, Nvidia, and Microsoft have successively released related product solutions. South Korea and Japan have also announced their vigorous deployment of the meta-universe track from the national level. The scenario depicted in "Ready Player One" seems to be a reality tomorrow.

(Source: Tsinghua University New Media Research

The Bloomberg industry research report predicts that Metaverse will reach a market size of US$800 billion in 2024, and PricewaterhouseCoopers predicts that the market size of Metaverse will reach US$1.5 trillion in 2030. The market potential is unlimited.

What is the meta universe? Simply put, the meta universe can be understood as a virtual world that is parallel to the real world and is always online. In this world, in addition to eating and sleeping need to be completed in reality, the rest including work, social interaction, entertainment, etc. can all be achieved in the virtual world.

Mysterious and unknown, the illusion that once only appeared in a dream may be realized in the near future, and it is exciting to think about it.

However, a reality that has to be admitted is that the high-fidelity virtual world has not been built, the image of people in the virtual world has not yet been established, the terminal cannot support the amount of data calculation, and the interactive experience is not good enough... The series of problems are waiting to be overcome, and the door to the meta universe has not been opened yet.

Two core technologies of the meta universe: virtual humans and real-time interaction

As mentioned above, the ultimate goal of the Internet-the meta universe, can break the space and time boundaries of human social activities. In a created virtual space, people from all over the world are engaged in real social activities, such as social interaction, business, entertainment...and the "virtual people" and "real-time interaction" capabilities are the two essential core technologies.

Virtual people represent an individual's recognizable image and identity, and real-time interaction capabilities can achieve immersive real social activities, and ultimately blur the boundary between virtual and reality.

Virtual humans in the meta-universe refer to virtual humans with a digital appearance. Unlike robots with physical entities, virtual humans rely on display devices to exist. Generally speaking, we divide virtual humans into two categories: one is virtual humans that can be driven by real humans, and the other is AI intelligent virtual humans with artificial intelligence.

The meta-universe is another infinite and vast space for real human social activities, so technically realizing virtual humans that can be driven by real humans is the first step into the "new world", of course, like the self-conscious NPC virtual in "Out of Control Player" If the characters can also be realized in the meta-universe, it will be infinite reverie.

(Movie "Out of Control Player")

Therefore, the virtual human of the meta universe should have the following three characteristics:

The first is the external image of the virtual person, which has a human appearance or cartoon and other interesting and vivid appearances, with specific appearance, gender, personality and other character characteristics;
The second is the expression ability of virtual humans, possessing human behavior, and the ability to express with language, facial expressions and body movements;
The third is the ability of virtual humans to perceive and interact, possess human thoughts, and have the ability to recognize the external environment and communicate and interact with humans.

Appearance, expression, perception, these abilities that almost every human being has, actually have high requirements for technology and equipment.

First of all, everyone has a heart for beauty. In real life, I am still a short distance from "beauty in the prosperous age". Maybe in the new world, I can have the opportunity of "If my life comes again" Qingcheng is heading to the pinnacle of life. But creating a "beautiful and good" avatar is not an easy task. The technical threshold of AI and images is very high. For example, 3D modeling, high computing power and rendering requirements for device performance are all difficult problems in the industry.

Secondly, with the prosperous beauty of "shen fish and geese, closed moon and shame flowers, all over the country and the city", just a static image, without facial expressions and no physical expressions, the world of the meta universe instantly becomes a world of "walking corpses". This involves voice interaction (TTS, ASR, NLP, etc.), animation synthesis (drive, rendering) and other AI-related technologies. The threshold for technical requirements can be imagined.

Finally, the meta-universe maps the real human world, so "interaction", the basic element of human social activities, is the most critical part of the construction of the meta-universe. In order to restore the real offline interactive experience without violation and sense, low latency, High-quality real-time communication services are guaranteed, but the current complex and changeable public network environment, terminal equipment and other factors pose a major challenge to communication transmission.

In order to be able to "fly freely" in the "new world", the avatar must be "beautiful", the expression must be "clear", and the communication and interaction must be "smooth"... Many problems need to be solved, but at present, the majority of Internet users still use mobile phones as mobile phones. The main and model types are also complex and diverse. Therefore, the solution that does not require external equipment, can solve the performance problems caused by the powerful computing power required to realize virtual human interaction, and at the same time guarantee the real-time interactive effect in a complex network environment, is currently the most practical and best approach." The best choice for "meta universe".

Netease Yunxin, here comes!

industry’s first virtual image real-time interactive integration SDK, taking the first step into the meta-universe

In response to the many problems currently facing, NetEase Yunxin and NetEase Fuxi Lab launched the industry's first "virtual image + RTC" integrated SDK, and based on this integrated SDK, a NetEase Yunxin virtual image real-time interactive solution was formed.

(Netease Yunxin virtual image real-time interactive solution)

This solution not only can vividly restore the image of the avatar, but also combines the real-time transmission capability of Netease Yunxin WE-CAN (Communications Acceleration Network), which can realize real-time interaction with the avatar, helping enterprise customers to achieve real-time interactive scenes of avatars with zero threshold. Take the first step towards the meta universe.

virtual image real-time interactive solution has six advantages:

1. Image: highly restored and extremely agile.

Netease Yunxin's real-time interactive virtual image solution can detect user facial expressions through cameras or uploaded videos, thereby driving 3D virtual characters to make the same expressions, including facial expressions, head posture, eye movements, tongue out, etc., which can restore tracking .

In order to reduce equipment performance requirements, some traditional methods often sacrifice user experience. For example, animation matching method: when the user speaks or acts, the frame animation is matched in the preset "emoji and action database", and finally A series of frames are matched and played to realize the "expression" of the virtual person. However, human behavior is diversified and random, and it is impossible to predict and presuppose all behaviors. It is conceivable that so-called "facial paralysis" or "zombies" will appear. The NetEase Yunxin avatar real-time interactive solution uses a "real-time capture on the end and real-time drive on the cloud" solution, which is more real and smart than the animation matching solution.

2. Hardware: It can be realized by mobile phone without wearing equipment.

NetEase Yunxin's avatar real-time interactive solution supports the use of ordinary monocular cameras for migration without other motion capture equipment, which is simple and convenient. Only ordinary mobile devices or PC devices need to install NetEase Yunxin SDK to generate and drive virtual humans, and interact with virtual humans driven by remote real humans in real time.

3. Performance: End-to-cloud collaboration, thousands of yuan machine can also be played smoothly.

After the user collects audio and video through the terminal device (mobile or PC), the action model data is output through the SDK of NetEase Yunxin, and the collected audio and video data are transmitted to the cloud for avatar reconstruction and synthesis.

Through the analysis, modeling and rendering of motion capture data in the cloud, NetEase Yunxin's virtual image real-time interactive solution greatly reduces the performance pressure of the dual-end algorithm, lowers the user entry threshold, and allows more thousands of yuan machine users to experience virtual interaction in advance The fun, feel the benefits of the meta universe.

4. Interaction: Low-latency, no lag, "face-to-face communication" in the meta-universe.

As a convergent communication cloud expert, NetEase Yunxin's RTC capabilities have always been in a leading position in the industry. Aiming at the indispensable "real-time interaction" scene in Metaverse, NetEase Yunxin WE-CAN global intelligent routing network escorts "zero distance" communication.

In the face of complex and diverse network environments and uneven terminal equipment, WE-CAN can stably provide real-time interaction capabilities with millisecond-level delays around the world, and select the best route through the intelligent routing network, reaching hundreds of locations around the world within 100 milliseconds Countries and regions provide audio and video services without stuttering for 99.9% of calls. Based on the highly reliable and low-latency NetEase Yunxin WE-CAN global intelligent routing network, Netease Yunxin's virtual image real-time interactive solution can realize real-time interaction of virtual people, just like face-to-face conversations in the real world.

V. Convenience: One SDK can realize the 2 core technologies of Metaverse.

Faced with the two major problems of avatar and real-time interaction in Metaverse, NetEase Yunxin’s integrated solution integrates avatar and RTC at the technical level and encapsulates it. Customers no longer need to connect with multiple suppliers, just one SDK. Construct a real-time interactive scene full of fantasy and high-experience virtual image

 
(Netease Yunxin "virtual image + RTC" integrated SDK function framework diagram)

NetEase Yunxin's "virtual image + RTC" fusion SDK undertakes light-weight tasks such as audio and video acquisition/pre-processing, data analysis, encoding transmission, decoding and rendering, etc., and transfers complex and high-calculation work to the cloud, which is formed with the help of the fusion SDK Efficient end-cloud collaborative work mode provides real-time interactive integration capabilities of virtual images. Only this SDK is needed to overcome the technical barriers.

6. Scenario: Applicable to multiple industries such as finance and e-commerce, reducing costs and increasing efficiency is one step faster.

It is worth mentioning that NetEase Yunxin's real-time interactive virtual image solution can not only be applied to the meta-universe world or the entertainment and social industries, but also can be widely used in all walks of life, helping enterprises to reduce costs and increase efficiency, while also improving customer experience. , Thereby increasing user retention rate and creating revenue.

1. Financial industry: virtual digital human customer service, providing 7 x 24 hours non-closing warm service

(application example in the financial industry)

2. E-commerce industry: virtual humans live live delivery of goods + customer service, bringing double improvements in business revenue and customer experience

(E-commerce industry application example)

......

Summary

Although entering the meta-universe requires solving one or another problem, it is undeniable that the realization of the meta-universe scene and the maturity of the meta-universe industry are only a matter of time. The scenes in "Ready Player One" may have to be imagined for a few more years, but the integration of virtuality and reality is a major trend in the development of the Internet.
For a long time, NetEase Yunxin keeps improving its technology, standing at the forefront of the industry and looking for the trend, hoping to help industry customers not miss every opportunity. Now, all friends and Netease Yunxin are welcome to take the first step to enter the meta-universe.

fill in NetEase Yunxin avatar real-time interactive program free experience information collection , try it for free now.

and reply "Meta Universe" to try it for free.