Decoding the Black Technology of Immersive Reporting of the Beijing Winter Olympics: Alibaba Cloud Video Cloud "Interactive Virtual Studio Technology"

This Winter Olympics, even if you are outside the snow and ice, you can feel the presence through the broadcast. This is what the interactive virtual studio technology brings.

"Virtual Studio" has entered the Olympic Village in the Tokyo Olympics, and in the 2022 Winter Olympics, its technical solution has been fully upgraded to achieve the broadcast and television-level directing effect. At the same time, the access threshold is greatly reduced, which greatly expands the application scenarios of this technology . At the same time, the RTC and GRTN networks are connected on the link, and the multi-camera and multi-view real-time synchronization technical capabilities are combined with the RTC real-time microphone connection and remote broadcasting capabilities, which can realize the superposition of multi-layer virtual backgrounds of anchors and guests, and enhance immersion. The live broadcast experience has been fully upgraded to a new level in terms of interactive effects, helping to create immersive reports of the Beijing Winter Olympics.

For the first time in the Centennial Olympics, interactive virtual broadcasts are used—creating immersive reports

Although the 2022 Beijing Winter Olympics is in the midst of ice and snow, the enthusiasm of the audience is like a song of ice and fire. We all look forward to hearing, seeing, and feeling the true voices, demeanor and conversation of the stars on the field after the game.

Therefore, as early as the pre-Winter Olympics, CGTN (China Global Television Network) was planning and producing a series of interviews and reports on the global Winter Olympics athletes and ice and snow stars, expecting to present the athletes' interviews in real time and wonderfully. Breakthrough is the best choice for CGTN to implement the Winter Olympics interview project.

Generally speaking, media interviews in different places need to be connected to the App, and the interview environment of athletes is very uncontrollable. Therefore, if you want to show a good effect of character interviews, you need to real-life keying and then composite them in the virtual live room, and at the same time, you must reach the end of the event. The real-time dissemination effect of the post-interview poses a great challenge to the technology. As a result, Alibaba Cloud Video Cloud's newly upgraded "interactive virtual studio" technology debuted at the 2022 Winter Olympics, solving this problem.

As a new real-time interactive virtual studio technology for cloud director, this is the first time that the Olympic Games has been launched. It has been fully applied by CGTN in the reporters' online reporting scene of the Beijing Winter Olympics, creating an immersive experience in reporting the Olympic events.

During the Winter Olympics, this technology assisted CGTN in conducting off-site interviews and virtual live broadcasts of real-life keying synthesis, creating a new form of off-site interactive broadcasting, realizing full-link production and broadcasting on the cloud, and creating a new model for traditional news program production. Brand new scene.

At the same time, through the real-time connection on the audio and video interactive APP, the real-time communication between the director and the reporter in different places is realized, and the news report is delivered to the audience at the first time. Through the full connection between the cloud director and RTC (audio and video communication), the online live room stream can be automatically obtained, and the news interview and live viewing can be synchronized. In addition, through the technical capabilities of the cloud real scene keying algorithm, the extremely fine keying synthesis effect is achieved.

This is the first time that the 100-year Olympic Games uses an "interactive virtual studio" for immersive event reporting, which has changed the form of interviews for traditional TV reporters. There is no longer a need for reporters to carry camera teams and heavy equipment to run around with heavy equipment. Respondents were set in the same-site solid-sealing mode. As long as the interactive virtual studio product of Alibaba Cloud Video Cloud is used, the interview task only requires "two mobile phones + one computer" to carry out live interviews and rebroadcasts anytime and anywhere, thus innovating the technical link of media production and dissemination and realizing the whole chain While the road production and broadcast are on the cloud, it also allows the audience to experience a more immersive broadcast viewing experience.

"Interactive Virtual Studio" schematic diagram

Three domestic first breakthrough super capabilities - the industry's cutting-edge technology

The ability to implement innovative solutions and create real-time interactive immersive event reports at the Winter Olympics stems from Alibaba Cloud Video Cloud's in-depth polishing of virtual studio products and three strategic technological breakthroughs:

The first pure cloud real scene keying confluence technology in China

This interactive virtual studio program is a new live-action keying algorithm and engine capability polished for the Winter Olympics. 4-channel keying source (green screen + real scene) is used for simultaneous real-time keying processing.

The value brought by many "firsts" is that it meets the requirements for live broadcast of the world's top events at the Winter Olympics level, and the real-life keying effect reaches the ceiling level in the industry.

In response to the program requirements of standard radio and television, Alibaba Cloud Video Cloud uses algorithms to break technical barriers. By trying various algorithms and engineering models, it breaks through itself again and again to improve the effect and performance of live keying. Under the uncontrollable and chaotic background of various interview environments, the technology can be handled perfectly, which greatly improves the experience of broadcasting content outside the Winter Olympics.

The first directing platform in China with "Seamless Connectivity between Device and Cloud"

In this Winter Olympics, the interactive virtual studio opened up the RTC and GRTN networks for the first time, combining the multi-camera and multi-view real-time synchronization technical capabilities with the RTC real-time microphone connection and remote broadcasting capabilities.

Under normal circumstances, the director station has always lacked end-to-end capabilities, so it is impossible to realize real-time interactive scenarios. Although it can also try to combine with various internal and external App applications, it is difficult to really implement it for various reasons.

This time, Alibaba Cloud Video Cloud RTC technology combines the director station to carry out in-depth transformation and linkage of virtual studio products, and fully optimize the interface effect and performance of the App side, providing ordinary users with simple and stable connection functions. Realize the "interactive virtual" studio capability.

The industry's broadcast and television-level directing capabilities

The brand-new interactive virtual studio fully meets the characteristics of ultra-low latency, ultra-stable, ultra-high concurrency, and achieves broadcast and television-level directing capabilities.

Ultra-low latency of the whole link from the Lianmai App to the director's confluence output, realizing the real-time communication of the Lianmai end and the "keying + composite rendering output" of the director station, all processes can be completed within 2-3S at the fastest. In the case of multiple people, multiple places, and long-term connection, the processing scene of multi-background matting can achieve the effect of stably processing zero interruption. The implementation of the entire solution can withstand the test of large-scale distribution and viewing, and is still stable and excellent for ultra-high concurrency.

"Interactive Virtual Studio" production full link

Technological upgrades have helped two Olympic Games successively—creating a new era of Olympic Games

At the Tokyo Winter Olympics, the virtual studio of Alibaba Cloud Video Cloud entered the Olympic Village. At that time, corporate representatives in Tokyo, Japan and Beijing, China successfully completed the immersive live broadcast connection through the cloud director's virtual studio and real-time subtitles.

2020 Tokyo Olympics virtual studio allows the two places to "co-exist" in the Olympic

For the 2022 Beijing Winter Olympics, in order to combine CGTN's reporters' interviews and reporting scenarios, and at the same time satisfy CCTV reporters' in-depth immersive reporting, Alibaba Cloud Video Cloud focused on the interactive virtual broadcast program and carried out in-depth upgrades in several aspects:

First, it focuses on creating a new algorithm engine and extreme effects for live keying, which solves problems such as rough edges and jitters in live keying. , to break the industry's real-life keying barriers.

Second, it enriches the access capability of cloud director on the scene side, realizes the complete solution of "real-time interaction + director", and creates a "real-time interaction" scene. In addition, the simple and convenient start of broadcasting through the App side makes the use threshold very high. Low, thus highly satisfying the use demands of TV reporters. The virtual studio strengthens and supplements the push-stream interaction link of the front link, further realizing an immersive and real-time live broadcast experience.

Third, comprehensively upgrade the stability of directing and broadcasting, ultra-long-term microphone connection stability, image keying stability, audio and video interactive App performance, etc., so as to fully meet the needs of remote, multi-person, multi-scene, and ultra-long-term microphone connection and live broadcast. , for complex multi-scenarios (single/double, long-range/close-range, live insert spacers) to achieve seamless switching, truly reaching the level of radio and television program production.

The comprehensive technology upgrade has upgraded the "virtual studio" to an "interactive virtual studio" product solution with real-time interactive and immersive effects, thus bringing a new experience of event reporting to the 2022 Winter Olympics.

More space for interactive virtual show—more scene value

The Beijing Winter Olympics is about to come to a close, and the interactive virtual broadcast will continue to support the real-time immersive online broadcast of the Paralympic Games. Super events like the Olympics have polished technical solutions to serve more than just that.

As a brand-new cloud director real-time interactive virtual studio technology, the interactive virtual studio can use the audio and video interactive app with low threshold to initiate live broadcast and real-time connection interaction. At the same time, based on the ability of cloud keying and virtual background synthesis, it provides " End-to-cloud-to-end” virtual studio solution to create an immersive live broadcast experience.

It is specially built for remote interaction to achieve an immersive experience in a virtual live room. In addition, it also provides solutions for remote production and remote interactive live broadcast scenarios. For example, athletes in ski resorts can have a virtual combined screen with the host of the news center and conduct real-time interactive interviews.

From the actual application field, it not only supports sports competitions, live broadcast of domestic and foreign events, remote commentary, multi-host mode and other scenarios, but also the interactive virtual studio can also be used in more industry scenarios, such as e-commerce and live broadcast of events Support, multi-mode broadcast of conferences, as well as various types of remote connections and virtual interviews, with a wide range of scenarios and rich capabilities, can be achieved.

All in all, with practice and precipitation in the 2022 Beijing Winter Olympics, Alibaba Cloud Video Cloud will further deepen the interactive virtual studio solution into a stronger product output, and provide remote broadcast and virtual live broadcast rooms for various industries involved in live broadcast scenarios. solution.

From the perspective of subsequent development, in terms of technology upgrades and product optimization, "interactive virtual studio" will continue to evolve and make breakthroughs in three directions:

First, after realizing low-threshold access, more emphasis is placed on achieving continuous cost reduction and improving the cost-effectiveness and competitiveness of the entire product; secondly, strengthening and improving the end-to-end layout and enriching the interactive capabilities of the director (such as the return of the broadcast screen from the audience, etc.), So as to provide a more complete interactive virtual studio program. In addition, under the general trend of shuttle construction between physical and virtual worlds, the concept of "virtual studio" will be deeply integrated to explore the combination and implementation of virtual human, 3D background and other capabilities to create more possibilities for virtual live broadcast rooms. .

In the future interactive virtual broadcast, the technology is more forward-looking, the threshold is lower, the scene is richer, the interaction is more diverse, and the effect is more immersive. Leveraging the light of the Winter Olympics, the benefits of technology are just around the corner.

"Video Cloud Technology", your most noteworthy public account of audio and video technology, pushes practical technical articles from the frontline of Alibaba Cloud every week, where you can communicate with first-class engineers in the audio and video field. Reply to [Technology] in the background of the official account, you can join the Alibaba Cloud video cloud product technology exchange group, discuss audio and video technology with industry leaders, and obtain more latest industry information.