Sun Yongnan: Real-time Video SDK Black Box Testing Architecture丨Dev for Dev Column

The full name of the Dev for Dev column is Developer for Developer. This column is a developer interactive innovation practice activity jointly initiated by Shengwang and the RTC developer community. Through various forms of technology sharing, communication and collision, and project co-construction from the perspective of engineers, the power of developers is gathered, the most valuable technical content and projects are mined and delivered, and the creativity of technology is fully released.

This article is a series of column content. The author is Sun Yongnan, an engineer at the Audio and Video Laboratory of Sound Network.

What is a test automation framework

A test automation framework is an infrastructure built to provide an execution environment for automated test cases or scripts. Automated testing frameworks provide users with various benefits that help them efficiently develop, execute, and report automated test cases. An automated testing framework is more like a system created specifically for automated testing. In a very simple language, it can also be said that a framework is a great fusion of various coding standards, testing procedures, work practices, project hierarchies, modularity, reporting mechanisms, test data injection, and other features that support automated testing.

Types of automated testing frameworks

Now that we have a basic understanding of automation frameworks, let's take a look at the various types of test automation frameworks that are popular today. These frameworks may differ from each other based on their support for automating different key factors such as type of driver, reusability, ease of maintenance, etc.

● Module-based automated testing framework

● Warehouse Architecture Automated Testing Framework

● Data-driven automated testing framework

● Keyword-driven automated testing framework

● Black box hybrid automated testing framework

● Behavior-driven automated testing framework

Why choose the black box hybrid automation testing framework to test the SDK

The so-called black box is provided to business testers without considering the internal structure and internal characteristics of the program, input test parameters and select output items in the program interface, and obtain the corresponding results through the internal mixed test framework of the program. Users only need to care about input and output.

The original intention of the scene design

"Automation is to better liberate hands and pursue higher efficiency"

Different from the test of Internet software (app, web), in short, the real-time video SDK test hardly needs a little bit, basically, the video communication between the end-to-end after the custom network damage is completed through the self-developed automation tool. , and collect the SDK log on the terminal as the test output data, and the objective test data is the objective test result. Focusing on this and combining with actual business needs, we can get rid of "Code based automation", and provide solutions based on platform and modularization of business tests, thereby providing more test dimensions, reducing repetitive manual labor and efficiency bottlenecks.

01 Solution Architecture Brief

infrastructure plan

● Adopt CI cluster + testing tools and automated testing framework + data platformization

Support daily, release testing, and development self-testing work

● Brief description of specific module functions

Real scene of landing computer room

● Multiple sets of test nodes support the entire video publishing business line

02 A brief introduction to thinking based on the dimension of automated testing

Gradually improve the automation closed loop

Usually, in the process of automated testing, we usually complete the step of "executing the test" first, but this is only a part of the relative automation. The advantages of closed-loop automation in my personal understanding are not limited to "convenient input, full test coverage, and test obstacle avoidance." Strong performance, high output aggregation", and more stand on the entire link to gradually break through the bottleneck of test accuracy and efficiency.

Here's an example from our testing of the obstacle avoidance and output aggregation modules:

Example 1 The impact of network fluctuations during time periods

When using chat software video in life, there is often a sudden freeze or blurred picture due to sudden network fluctuations. The fluctuation range and time are uncertain, and the test of the real-time video SDK will also encounter such problems. Although we try our best to ensure the stability of the network environment, we often encounter such problems during long-term testing, which affect our test data.

How to reduce data errors caused by network fluctuations during testing?

● Use the funnel-type rerun screening method, the brief structure is shown in the figure

That is, the cyclic evaluation ensures that the test data of the SDK version is effectively reduced within the set error because network fluctuations affect the SDK version.

Example 2 Version data fluctuation influence adoption

After the automated test is completed, the data of various experience indicators between the test version or the release version is generally based on the difference in the numbers between the reports, but with the increase of experience indicators, we often more urgently need multiple versions to be accurate to end-to-end Visualization of differential perceived quality on one of the metrics above.

● Background management system - objective reporting module

Dynamic view of evaluation of cases, devices, experience indicators, etc. that supports multi-version report comparison

At present, although this view form of objective report temporarily satisfies our need for forensic comparison on the data of automated reporting indicators, the data sorting and synthesis functions still need to be more deeply understood in the business.

03 Summary

There is no official standard for the framework of different businesses, but it is closely related to the actual business needs, and is integrated with the mainstream framework or self-developed framework with high applicability into the entire test framework to provide solutions. The operator may be used for development or testing, and everyone's technical lines are different. As a relatively "black box" user, you don't need to know his principle structure for the time being. You only need to clearly understand the problems that can solve the work for yourself in the shortest time. The value of black box testing frameworks.