A new way of video interaction that breaks the virtual boundary, the application concept and exploration practice of AR space writing

AR space writing demo

With the development of technology and the drive of the hyper-video era, the forms of interaction are becoming more and more abundant. From screen touch, to voice interaction, face, fingerprint, voiceprint, to AR and VR that have become popular in recent years... Humans have been accustomed to using the almost instinctive communication method of body and gesture to communicate long before the appearance of language. As the most basic and natural interaction method, there are more and more application scenarios of gesture interaction.

At present, the gesture interaction logic of most video applications on the market is mainly to trigger a preset single special effect through a specific gesture. This relatively simple interaction not only cannot exert the potential of human flexible palms, but also has a recognition effect on the terminal. Large room for improvement.

Especially affected by the epidemic and the huge demand for audio and video conferences and collaborative office today, it is very difficult to use a physical whiteboard to draw and write for remote communication and collaboration.

Although there are products similar to virtual whiteboards on the market, these products mainly rely on the mouse and other devices for input. We can use the natural advantages of gestures to replace the mouse, keyboard, touch screen and other interactive methods to realize AR space writing, It exerts its great value in office, life and entertainment scenes.

AR writing in space breaks the barriers of virtual whiteboards

How to realize a perfect virtual whiteboard through AR space writing?

The most direct idea is to render the written content on the screen. For example, a recent popular open source project "Yoha" achieved the effect through this idea, but it also faces the problem that the characters cannot be written very small due to the limited viewing angle of the camera. and limited writing content.

Another solution is to write a part of the content first, shrink it down, and then write another part of the content. This solution seems feasible, but suffers from typographical difficulties and poor continuity of content before and after.

The AR space writing capability of Alibaba Cloud Video Cloud Beauty Effects SDK (hereinafter referred to as "beauty SDK") allows the AR space writing window to be freely enlarged and reduced by suspending the AR space writing window on the virtual whiteboard. , pan, so that the user can freely control the size and position of the writing, and the typesetting of the writing will be more controllable.

The edge of each frame of the image captured by the camera is cropped, and then suspended on the whiteboard. The user can zoom in or zoom out the ROI window to control the size and fineness of the written content.

Users can also control the writing position by moving the AR air writing window.

When the user's gesture (virtual pen tip) moves near the edge of the AR window, the AR window will automatically move in the corresponding direction (refer to the moving windows of games such as DOTA, LOL, and Warcraft).

Referring to the moving picture, this operation method that does not need to move the body not only conforms to people's writing and usage habits, but also greatly improves the convenience and comfort of moving the window.

Alibaba Cloud Video Cloud integrates the AR space writing capability as a "hidden black technology" into Dingding's audio and video conferencing hardware products. This capability can help participants communicate through space writing or drawing during remote meetings. . At the recent DingTalk conference, Alibaba Cloud Video Cloud also interactively demonstrated this capability.

https://www.youku.com/video/XNTg2MjI1NjA5Ng==

DingTalk 2022 online conference, live demonstration of AR space writing

Rich virtual special effects to make video interaction more interesting

AR space writing can also be combined with particle special effects to display various rich and cool special effects such as snowflakes, flames, water droplets, petals, smoke, etc., providing users with room for personalized creation and making video interaction more beautiful and interesting.

The AR space writing ability has recently been launched in the beauty SDK of Alibaba Cloud Video Cloud. This is based on the self-developed facial key point technology, which supports image beautification, portrait beauty, image keying, sticker beauty, motion recognition, smart fun A variety of personalized customized beauty interactive services such as interaction and keying processing.

Meixiao SDK has multi-dimensional advantages:

• Good effect: full-featured, one-key combination and item-by-item DIY
• Small package body: the basic beauty function only needs 0.78M
• Excellent performance: Android at least supports 4.3 system, iOS system at least supports iOS-8 system, Mac supports the latest M1 system
• Fast and customizable access: independent assembly and disassembly, parameter-level adjustment and customization on demand

Based on a series of application advantages, Meixiao SDK is suitable for various business scenarios such as live broadcast, shooting, conference, e-commerce, etc., which perfectly balances the problem of effect beautification and performance overhead, and makes video interaction richer and more interesting.

It is foreseeable that gesture interaction is an indispensable part of human-computer interaction in the future, a light and borderless immersive virtual world, it is impossible to completely rely on "handheld devices" and physical "contact interaction", and only use technology to free your hands. It is the right way to open the seamless link between virtual and reality.

The interaction bottleneck of video-based scenes has begun to appear. The development and application of the AR space writing capability of Alibaba Cloud Video Cloud based on the Beauty Effect SDK provides more possibilities for intelligent and interesting new interactions in the hyper-video era, and promotes the development of video interaction. Far.

Readers who want to experience AR writing Demo or communicate in the air are welcome to Dingding search group number: 34197869, or scan the QR code below to join

"Video Cloud Technology", your most noteworthy public account of audio and video technology, pushes practical technical articles from the frontline of Alibaba Cloud every week, where you can communicate with first-class engineers in the audio and video field. Reply to [Technology] in the background of the official account, you can join the Alibaba Cloud video cloud product technology exchange group, discuss audio and video technology with industry leaders, and obtain more latest industry information.