Human visual neuroscience helps the audio and video industry revolution-extreme real-time communication under weak network

1. What is a weak network?

1.1 Weak network concept

A weak network literally means that the network is relatively weak. We generally call it poor signal and slow network speed. With the rapid development of mobile Internet in these years, a large number of users will use mobile apps in special scenarios such as subways, tunnels, elevators, and garages. . In these scenarios, the network will experience delays, interruptions, jitter, timeouts, etc.

1.2 Network form

The network form includes wired connection, 2G/3G/4G/5G/Edge/Wifi and other network connection forms. From the perspective of testing, it also includes disconnection, network failure, etc., for the data definition of weak network, different applications The meaning of the definition is also different and unclear. Generally speaking, the rate below 2G is a weak network, and 3G can also be classified as a weak network. In addition, very low bandwidth <50kbps, weak signal Wifi, etc. It is also a weak network.

1.3 Research background

There are some special scenarios, such as: forest disaster relief, border surveillance, and other scenarios. These scenarios are often related to national security and life safety, and require strict real-time communication, but the base stations that these scenarios rely on are often interfered by natural factors, such as earthquakes. And other natural disasters.

Second, what technical attempts have you tried?

2.1 AI control

During watching the live broadcast, I heard that teacher Ma proposed a new concept. When the human eye perceives the image, the processing is about 100B/s, and then after the cells on the retina are separated, it is compressed by about 100 times, and then after a process The series of cell processing, in the end, is only about 40b/s, and the resolution of the area that the human eye pays attention to is relatively high, and the relative resolution of the area that the human eye does not pay attention to is a little lower. And for some areas, the human eye has special colors Sensitivity is called the attention mechanism.

Traditional flow control technology is often unable to select suitable algorithms and bit rate control according to the specific network environment during audio and video encoding and transmission. The AI control module (equivalent to the human brain) will collect video session experience (concerned by the human eye). Things), including the video encoder, the encoding status of the receiving end, the network, and the playback status. According to these characteristics, it can fight against network fluctuations and make coding parameter setting decisions.

2.2 Strengthen active network decision-making (compression and integration)

According to different users, that is, the playback terminal performs a personalized frame drop, but the overall perception will not be very different. This technology uses the principle of multi-frame video time and space consistency, based on the human cells for different images The feature sensitivity is different. Some cells are sensitive to color, some cells are sensitive to movement, some cells are more sensitive to directionality, and some cells are more sensitive to texture, so the human brain is sensitive to the audio and video information it perceives. It is not decoded by one bit or one bit like a decoder, but partially decoded. Therefore, for any video input structure, it is mainly divided into two parts, one part is used to store the texture details in the storage space, and the other part It is not so sensitive to the details of the movement, so the other way can not take up so much space. Of course, in the process of integration and reconstruction, intelligent learning is needed for compensation and transformation. Therefore, the final output audio and video will not feel very different.

2.3 Video bit rate adaptation based on reinforcement learning

According to video classification and network classification, online learning model training is carried out. For example, most boys like game videos, most girls like Taobao shopping videos, and the video bit rate and accuracy of different classification videos are different. Based on this, it is proposed whether For model training on different types of videos, the user terminal will select different algorithms when playing different types of videos. Based on the online learning platform, the efficiency of the offline model has been improved to a certain extent.

Three, personal perception

3.1 What are the specific application scenarios for weak network environments (1 Medicine Net/Chongqing 120 First Aid)

1 During the epidemic, the drug network urgently opened up a free online consultation channel for Wuhan, and expanded the scope to the entire territory of Hubei Province. Video technology, in the scene of video consultation, because doctors and patients are in different network environments, the above-mentioned weak network environments may appear. In these environments, Agora has excellent weak network transmission and anti-loss. The packet algorithm can still guarantee the smooth audio and video in the case of 60% packet loss, and the smooth voice of the network environment with 70% packet loss.

120 First aid is through video remote guidance + first aid teaching video guidance, which truly gains opportunities and time for life. But the same patients may be in a weak network environment, how to ensure the quality of audio and video transmission is still particularly important. Moreover, first aid is more about race against time, and the connection rate must be ensured. Failure to connect may mean delay in first aid. According to the official website data, the sound network has more than 200 data centers around the world. Based on this, the software-defined real-time network is built. In a poor environment, stable and reliable, high-quality transmission and a high connection rate of 99.9% can also be guaranteed.

3.2 Experience

The business form is changing, and technology must keep up. I originally thought that with the continuous development and progress of technology, such as 5G, GPU, chip and other hardware equipment upgrades, for software developers, network jitter or hardware environment can be ignored. It is more difficult to think about whether there will be a day when the software developed by yourself may need to run in a more demanding environment, or the services provided, the equipment used by the user is too old and incompatible, etc., so it is usually very I don't pay attention to the robustness of the code, just use it, and make do with it. These habits have unknowingly and subtly affecting me. I don't know if any classmates are like me.

In the past, the concept of audio and video has been stuck in the more traditional codec, live streaming, video on demand and other common applications, and there is no deep thinking about the differentiation of the network environment where each user is located; therefore, the study of extreme video under weak networks Communication is not fault-finding, it has very important practical significance, ranging from national defense and security to all aspects of people's lives.

Under the trend of artificial intelligence, combined with AI, and human visual neuroscience, the audio and video fields can also borrow from the wind to seek technological breakthroughs and innovations. In addition, I personally believe that the rise and application of concepts such as edge computing and fog computing have shortened the distance between users and services. In the past, services were deployed in central nodes. Now deployment in the form of microservices will be more efficient, such as WebRTC services to edge nodes. In addition, the deployment service cost of edge nodes is lower and bandwidth is saved.