Qcon Global Software Development Conference Rongyun shares SDK delivery quality assurance experience

On May 29-31, 2021, the comprehensive technology event QCon Global Software Development Conference hosted by InfoQ China, a subsidiary of Geekbang Technology, was held in Beijing. The conference settings include "artificial intelligence-driven business practices", "cloud native technology applications", "big data real-time calculation and analysis", "audio and video experience improvement actual combat", "low code practice and application", "flexible engineering and operation and maintenance" In 35 technical topics including "System", the industry’s top technical experts are invited to share technical innovations and practices with technical team leaders, architects, engineering directors, and developers.

Picture 1: 2021Qcon·Beijing Station

In the audio and video service solution special session, Rongyun, a leading global Internet communication cloud manufacturer, was invited as an invited guest to focus on the exploration and practice of the automated test link of Rongyun audio and video SDK, allowing developers to understand the SDK package integration test process Difficulties and pain points, as well as the design ideas of diversified SDK automated interactive testing.

The ultimate goal of testing is to ensure the quality of the SDK

Since its establishment in 2014, Rongyun has focused on the communication cloud track, providing the SDK/API to provide communication cloud capabilities for hundreds of application scenarios. In order to continuously meet the needs of developers for product functions, Rongyun's SDK interface number has increased from 279 to 597 in the past 7 years, and the number of server API interfaces has increased from 36 to 107, and it is compatible with Android, iOS, and Web. , Flutter, applet, desktop version and other platforms.

At the same time, Rongyun also integrates numerous SDKs into different types of demos, which are convenient for developers to call, covering the live broadcast mode SealLive, instant messaging SealTalk, Rongyun classroom SealClass, real-time audio and video SealRTC, voice chat room SealMic, video conference SealMeeting, etc. Many scenes. These scene-based Demos and SDKs have accumulated more than 300,000 apps, and the SDK has reached 5 billion users.

With the increase in the number of SDKs and the diversification of demo scenarios, developers are most concerned about how Rongyun guarantees the quality of terminal SDKs. At the meeting, Rongyun Automation Test Architect Chen Peijie delivered a keynote speech on "Exploration and Application of Rongyun SDK Delivery Quality Assurance", starting from actual combat experience and answering questions for developers.

Figure 2: Rongyun automation test architect Chen Peijie delivered a speech

At present, there are many indicators for measuring the quality of audio and video, which are divided into two categories: those directly related to audio and video and those related to the network. Among them, directly related to audio and video include video connection rate, stuttering, sharpness, blurred screen, black screen, first frame time, as well as audio howling suppression, echo cancellation, noise suppression, etc.; while in network transmission, weak network, loss Packet, delay, jitter, and network adaptive adjustments will also affect the quality of the terminal SDK. In addition, performance indicators such as CPU, memory, IO, flow, heat generation, static code scanning related to security and compatibility, security tool analysis, version compatibility, and device fragmentation are all factors that affect the quality of the terminal SDK.

Therefore, in order to deliver a quality-assured SDK, Rongyun's approach is two words: testing. Only through testing can we grasp these indicators, and then optimize the weak links to ensure the stability of the SDK. In the testing process, Rongyun realizes the automation of Demo business testing through tooling and scripting, and through rapid iteration of business testing, special testing, and SDK interface testing to follow up and master relevant indicator data.

Practice of specialized automated testing based on business

Rongyun's thinking on the automated testing of the Demo business is based on two dimensions: one is how to achieve automated business testing to ensure the diversity of terminal SDK products and the stability of multi-terminal interaction? The other is how the audio and video special test is carried out?

In terms of business automation test practice, Rongyun has developed a scientific terminal business automation test process through the selection of terminal business automation test tools.

Figure 3: Automated testing process for terminal services

Through this process, it is possible to first realize any free combination among a large number of Android, Web, iOS, Win, and Mac, for multi-terminal interaction and cross-testing. Second, use plug-ins to focus on important quality indicators such as connectivity rate, stability, screen detection (black screen, green screen, etc.), performance data, weak network, and packet capture data. Third, the terminal process data is completely preserved. For example: log, audio after real terminal rendering, video recording, etc.

In the practice of audio and video special testing, the test needs to achieve end-to-end overall coverage, including collection, preprocessing, encoding, network transmission, and then decoding, post-processing and rendering the entire transmission process.

Figure 4: Audio and video special test practice

To complete this process, Chen Peijie introduced the main steps of the audio and video anti-packet loss automated test as an example:

First, link to the network loss meter route through the business UI automation platform;

Second, the relevant parameters of the network loss meter are called from the interface and transmitted to the terminal device, and the terminal screen is recorded and uploaded to the platform.

Third, automatic or manual analysis to complete the automatic test of audio and video anti-packet loss.

In addition, Rongyun can also conduct automated testing through the combination of traditional manual + testing tools, complete end-to-end delay testing, and implement performance data statistics through tools.

Through these tests, we can finally master related performance indicators such as CPU usage, memory usage, power consumption, flow, FPS, freeze, etc., as well as video and sound delay; video and sound quality; howling distance, howling suppression and Audio and video call anti-packet loss test data.

Rongyun SDK interface automated test exploration

The so-called SDK interface automated test is to verify the coverage of SDK interface field verification, the testability of complex calls to the SDK interface, and the coverage of system version and device fragmentation. Currently, there are three commonly used SDK interface testing solutions: SDK unit testing, UI automation methods, and registration broadcast methods.

Figure 5: SDK interface test plan

In practice, Rongyun believes that SDK automated testing tools should have the following capabilities:

First, most of the SDK interfaces can be quickly iterated away from the UI;

Second, it is possible to perform free combination logic verification on the SDK interface fields;

Third, SDK package integration tests can be carried out in a wide range.

Based on this, Rongyun has carried out two explorations and iterations of the SDK automated test tool. The first version adopts the terminal device microservice model, and the SDK interface is bound to the URL address; the second version implements the parameters through the URL dynamic reflection interface. Dynamic reflection serialization.

Finally, Chen Peijie emphasized: automated testing should achieve higher coverage in the shortest test cycle, and automated testing should not be process-centric as much as possible, and should adapt to the needs of rapid changes in the scene.

Figure 6: Developer listening to the speech

Concluding remarks

Through testing and optimization, at present, the maximum anti-packet loss of Rongyun audio can reach 80%, and the maximum anti-packet loss of video is 40%; the delay of voice is less than 120 ms, the delay of video is less than 200 ms, the delay of voice live broadcast is less than 350 ms, and the delay of video live broadcast is less than 350 ms, which can guarantee real-time interaction without perceptual delay between end-to-end. It can also be seen from Chen Peijie’s sharing that Rongyun has excellent audio and video capabilities due to Rongyun’s stringent requirements and unremitting exploration of various technologies. Excellent indicators and excellent quality will always be the direction that Rongyun pursues.