How to conduct automated audio and video testing for different regions overseas?丨Dev for Dev column

Due to the global COVID-19 pandemic, the demand for real-time audio and video around the world has skyrocketed in recent years. Different countries and regions have different network environments due to economic development, national policies and other reasons. If you want to have a good audio and video experience, you need to test audio and video indicators by region. But neither outsourcing nor cloud testing can meet our quality requirements.

This article will introduce how Shengwang conducts audio and video automated tests in different overseas regions under the current new crown epidemic, and obtains reliable indicator results.

This article is the content of the "Dev for Dev Column" series. The author is Hu Dahua, an Android development engineer at Shengwang Audio and Video Lab.

01 Traditional audio and video test methods are no longer applicable

Taking video delay measurement as an example, our usual practice in the past is: first find a network clock, then let the sender and receiver make a video call, and use the sender phone to shoot the clock, and then the receiver will see A picture of the network clock. We subtract the time of the network clock from the clock time displayed by the receiver's mobile phone, which is the delay of this frame of video. As shown in Figure 1.1:

■Figure 1.1 The current frame delay is 315ms

This test method requires testers to go to the site to arrange the test equipment. However, in the current epidemic environment, it is difficult to send employees to overseas business trips for field testing.

02 Why not outsource to an overseas testing team?

You may think that since you can’t send employees to travel overseas, can you ask local people for help, or outsource to a local professional testing team?

We have also considered this strategy. However, audio and video testing is different from general software black-box testing. There are many test cases in the testing process, and each use case needs to adjust different parameters. It is difficult for external test teams to pay attention to details in our usual testing. In addition, they do not Possess the expertise required to test audio and video. There is no guarantee that test results are accurate and reliable. Furthermore, due to language and time zones and other reasons in different countries and regions, the cost of coordination is extremely high.

03 Automated testing with cloud testing

We try to use mobile phones deployed by cloud testing providers in different regions overseas for testing, install test programs on these mobile phones, and control mobile phones through remote desktop or automated scripts to make audio and video calls in China.

Large cloud testing manufacturers such as Headspin have deployed cloud testing mobile phones in dozens of foreign countries and regions, but cloud testing mobile phones are different from real phones, and have many limitations: For example, if the camera is covered, it is impossible to use those captured by the camera for video transmission. of test cases. Therefore, we need to design a solution that does not use cameras to measure audio and video indicators. We thought of testing video metrics by self-collecting YUV video.

Using self-collection YUV, video transmission between two cloud-tested mobile phones is realized. How can we get the performance indicators of video transmission? Such as delay, freeze, bit rate, frame rate, etc. Using the YUV self-collection method, there is no independent clock source to refer to, and the time-synchronization problem of two mobile phones must be solved to measure the delay. We try to synchronize the time through an NTP server or a local area network. If both mobile phones are under the same local area network, the time synchronization through the local area network will be very accurate. We measure the round-trip delay rtt<10ms of data packets exchanged between the two mobile phones locally, and the uplink and downlink speeds are the same in the same local area network. , then the actual timing deviation should be <2ms. This meets our requirements for accuracy.

However, some cloud testing providers cannot communicate via LAN between two mobile phones, such as Headspin. At this time, we consider using NTP to synchronize the time. The time error of NTP is tens of milliseconds, which is much larger than that of the local area network. If it exceeds 50ms, it will greatly affect our measurement delay. We hope that the time error is less than 50ms. How do i do this? By consulting the NTP official documents, we know that as long as the time rtt is small enough, it can be achieved. The relationship between rtt and time offset is shown in Figure 1.2 below (from the NTP official website):

■Figure 1.2 Relationship between rtt and time synchronization deviation in NTP time synchronization

As can be seen from the above, as long as the round-trip delay rtt is controlled within 100ms, the time deviation is -10ms<offset<10ms, so the time deviation of the two mobile phones will not exceed 20ms, which meets our requirements. Can rtt<100ms be achieved in the actual environment? Through actual testing, we found that it can be done. When paired through Alibaba Cloud's NTP server (ntp.aliyun.com) in Shanghai, the rtt is around 30ms, rarely exceeding 60ms.

Overseas, taking New Delhi, India as an example, the rtt of the local NTP server (time.nplindia.org) is also easily below 30ms. Therefore, as long as we use the local or relatively close NTP server to synchronize the time, the accuracy can meet the requirements. When the program is implemented, we can adjust the time multiple times, and take the minimum value when rtt<100ms.

After solving the time synchronization problem, you need to know the sending time and the receiving time of the video frame. We can draw the sending time on the video frame at the sending end, and the receiving end will overlay the current time on the received video frame in the form of a floating window, and record the screen of the receiving end. After the time is recognized by AI, the delay of this frame can be calculated. As shown in Figure 1.3:

■Figure 1.3 Time-stamped video transmission

In addition to the delay, the next step is to calculate the frame rate, bit rate and stall rate.

The calculation of the frame rate is relatively simple. Take each frame of the video recorded by the receiving end. If the number on this frame changes compared to the previous frame, we consider it to be a different frame. The calculated frame number divided by the time is For the calculation of frame rate and bit rate, we can use Wireshark to analyze by capturing packets. In audio and video calls, video data is transmitted by UDP protocol, which can be clearly distinguished by Wireshark.

The freezing rate is related to time. Usually, we calculate the freezing rate in the cases of 200ms, 300ms, and 600ms. The calculation is relatively simple. Taking 200ms as an example, as long as the interval between different frames before and after exceeds 200ms, it is considered to be stuck. , the number of stalls divided by the total number of frames is the stall rate.

04 Other test methods

Compared with cloud testing, is there a simpler test solution? We thought that with the help of the SDK's own statistical information, RTC solution providers will integrate detailed and comprehensive statistical information in the SDK for external use. We can pass the callback form. get. For example, the audio and video indicator information of the current call can be obtained through the onRtcStats (IRtcEngineEventHandler.RtcStats stats) callback of SoundNet. For more detailed information, you can visit the SoundNet Documentation Center (docs.agora.io/cn) to search for it. Of course, this method is a reference when other solutions are not available.

The above are just the words of one company. There are many solutions for audio and video testing. Maybe you have a more accurate and efficient method. Welcome to share it in the sound network developer community rtcdeveloper.agora.io, let's communicate and make progress together!

About Dev for Dev

The full name of the Dev for Dev column is Developer for Developer. This column is a developer interactive innovation practice activity jointly initiated by Shengwang and the RTC developer community.

Through various forms of technology sharing, communication and collision, and project co-construction from the perspective of engineers, the power of developers is gathered, the most valuable technical content and projects are mined and delivered, and the creativity of technology is fully released.