Software testing from 0 to 1

foreword

Wu Rang is a senior solution architect of GitLab, with 10 years of experience in the industrial Internet industry. Worked as architect and technical director in startups and listed companies successively. Alibaba Cloud MVP, AWS SAP, CSM, TOGAF certified architect. In recent years, it has been committed to the digital transformation of enterprises. Now responsible for the design and promotion of GitLab's solutions.

This article is based on Wu Rang's reorganization of the content shared by Wu Rang in the " Soundnet Developer Entrepreneurship Lecture • Phase III丨How to Ensure Product Quality in the Early Stage of Entrepreneurship ". Follow the official account "Shengwang Developer " and reply to the keyword " JT0611 " to download PPT materials related to the event.

01 Theory

1. Project Mercury and software testing

Speaking of software testing, it is said that the earliest recorded software testing was mentioned in the first manned spaceflight program of mankind, that is, the Mercury program. I am not sure if it is the earliest, but I did find some records. Because manned spaceflight involves life safety and political factors (the Cold War space race between the United States and the former Soviet Union in 1958), the United States attaches great importance to it and introduced software testing, so you can also see why many companies do not pay much attention to testing. In fact, It is because the product does not involve life safety or politics, and everyone has not encountered these so-called red lines, so they do not pay so much attention to this part.

2. Development history of software testing

Figure 1 shows the development history of software testing, which can be initially divided into five stages. It started in the 1950s at the earliest. At this time, there is no testing, and more is called debugging. It wasn't until 1957 that Charles baker came up with the concept of testing.

The difference between the so-called testing and debugging is that the purpose of debugging is to make the program meet the expectations of the developer, while for testing, it is necessary to confirm whether the program meets the expectations of the functional requirements of the product. The third stage is not only to do tests, but also to conduct some exploratory tests on the program to confirm whether there are things that should not be done. The fourth stage is the V&V theory that testers are familiar with, which corresponds to the V model in software testing. This model is still widely used in the entire industrial field. It is the first time that software testing should be applied to the entire software. in the life cycle. I think that as of today, the entire software testing is still in the fifth stage, which is to do some exploratory testing, agile testing, and the so-called automation tools brought about by the rise of new concepts such as TDD and BDD. Continuous integration technology Applications. So throughout Phase 5, our aim is to prevent bugs at the code level, not just confine testing. These five stages are not completely independent, more of an inheritance.

■Figure 1

3. Software testing classification

The upper left corner of Figure 2 is the traditional software classification, which may be familiar to those who do the testing. It is actually a classification of software testing methods according to different dimensions. However, this so-called traditional classification is prone to the intersection of concepts and the problem of blurred boundaries, which makes this set of classification theories more complicated.

■Figure 2

For start-up enterprises, if there is such a complex and vague conceptual system within the enterprise, it is not conducive to communication, and it is not easy to improve the unity of cognition, or directly affect the efficiency. In order to solve such a problem, there is the test pyramid in the upper right corner. This pyramid integrates all kinds of ambiguous test methods, forming a three-stage or multi-stage type, from the bottom to the top, from the big change. small process. The layers here include unit layer, service layer and UI layer. The further down, the higher the independence; the higher up, the higher the integration. Therefore, you should write more test cases in the lower layer and do less tests in the upper layer. This is a healthy test state.

Coincidentally, Google has also re-classified testing in "Google Software Testing". Its classification method is divided into large, medium and small tests. In a sense, this division is similar to the testing pyramid. Correspondingly, the mini-test corresponds to the unit level in the pyramid test, which means that some individual modules can be tested, and the mini-test can handle independent functions. At the same time, it also gives a proportion. Google believes that the so-called small test should account for 70%, the medium test should account for 20% (the integration test between modules accounts for 20%), and the final comprehensive systematic test only needs to account for 10% . Below Figure 2 is the classification given by Google. For start-up companies, it is recommended to choose the test pyramid model. The number of layers and the name of the company can be customized. However, for such a model, the concept must be unified first. Otherwise, if the language of testers and developers is not unified, then in the There are many problems when working.

4. Automated testing tools

Automated testing has developed vigorously along with concepts such as agile development and agile testing. Through Figure 3, you can see the mainstream automated testing tools. In the early stage, it was mainly linear testing, that is, scripts and macros were generated by recording. This is a comparison Common form, post-testing tools have more types of changes, such as support for structured, data-driven, and keyword-driven testing. There are two main forms in terms of form, one is based on UI, the other is based on API, the form based on API is to call API, and more is to operate at the interface level. The UI-based form is to simulate the user's operations, such as moving the mouse, clicking the mouse, and buttons on the interface.

■Figure 3

5. Conditions for automated testing

I think there are two main advantages of automated testing. The first point is that it can replace some repetitive but necessary testing work and solve the problem of repetitive labor. The second point is also very important, because repetitive work is easy to introduce human errors, and tools should be used to assist at this time. Although the advantages of automated testing are obvious, its disadvantages should not be underestimated. Many people think that automated testing is a panacea and can be carried out in any situation. In fact, it is not. Its disadvantages also include two points. The first point is the high cost. , especially the short-term overhead is relatively large. If automated testing is to be performed, talent training or talent recruitment, process establishment, script writing and maintenance are required, all of which require a relatively high cost. In addition, as mentioned earlier, the advantage of automated testing is that it can replace repetitive work. The corresponding disadvantage is that it is not easy to find new bugs, because it can only do repetitive tasks. It is difficult to find new bugs. Although there is fuzzing and chaos engineering now, it is still not as good as people or able to find new bugs.

Under what circumstances, start-ups can carry out automated testing, there are three main points: the first point is that the product cycle is relatively long, here refers to stable products, if it is a demo product, there is no need to automate it. The second point is frequent iteration, which is where its value lies. The third point is that the product is relatively stable, which is actually tied to the first point in a sense. Only because the product is stable can there be stable income and investment.

So for automated testing, it is more of a complement to traditional manual testing, not a replacement. There are two aspects of complementarity. The first aspect is that if you do not have the conditions required for automated testing, you can do manual testing; on the other hand, it is not easy to find new bugs in automated testing. Then turn this part into an automated script.

6. Agile and software testing

Like agile development, agile testing is a part of agile development, it is an idea, not a method. So what is the difference between this idea and traditional testing? I think it mainly includes the following points:

The first and most important thing is communication, to carry out agile testing, and the most important thing is to test all staff for customer needs. From product, project, development to testing, it is necessary to clearly understand the needs of users. If relying on traditional documentation methods, the message The delivery efficiency is relatively low and not timely, which is the limitation of traditional testing. Therefore, for start-up enterprises, even if they do not adopt agile testing at the beginning, they can learn from its ideas to help solve the phenomenon that testers do not understand the requirements.

The second point is that agile embraces change, but embracing change is not about making people change requirements recklessly and frequently. Because agile always emphasizes slow thinking and fast action, it is not agile to rush forward with a slap on the forehead, so once the test is pre-positioned, it will be a large investment. At this time, the management of the company must think and control the demand. Changes cannot be made overnight, otherwise it is very easy to hurt the entire team.

7. Agile Testing Best Practices

The aforementioned agile is an idea, not a concrete practice. Although there are many manufacturers (including capital institutions) on the market who have mentioned the best practices, I think it is a general model. The situation of each enterprise is different. Applying a unified approach may have the opposite effect. .

Here are a few points to share with you. In the standard model, it is mentioned that the test moves forward, and the test should participate in the evaluation process of the requirements, as shown in the upper left corner of Figure 4. It shows that before the iteration starts, the tester will participate in the iteration. Test analysis, this analysis mainly refers to the disassembly and refinement of requirements, and then write content such as AC. The figure also mentions full regression testing, which is the characteristic of agile testing and automated testing, that is, full regression testing. Tests are required in each iteration cycle, and the previously tested content is re-tested, which requires continuous improvement and accumulation of automated test scripts. This is where agile and automated testing come into full play.

■Figure 4

As can be seen from the bottom of the figure, enterprises must practice agile testing as needed and gradually improve. For a small team, such as the unstable stage of a start-up with a team of less than five people, it is better not to apply automated testing all at the beginning. Our goal is to be simple and efficient, and too much content will delay it. However, it is still necessary to have such capabilities, because it is always ready to do large-scale and formalized business.

8. DevOps and software testing

Now many enterprises are doing DevOps transformation, so what is the relationship between DevOps and software testing? First of all, you can look at the lower left corner of Figure 5. The word DevOps here is not used well. It just takes the words of developers and operation and maintenance personnel into it, but in fact DevOps is software development, testing and operation and maintenance. Three aspects, this Words just embody two aspects. Testing is an important part of DevOps, as shown in the lower right corner of the figure. The difference between DevOps and agile development is mentioned. The most important thing is that the feedback cycle is faster. If testing is still done in the traditional way, then Its capacity and frequency are not enough to meet such a fast delivery frequency, which is a very big challenge for testing, and it is also a reflection of its importance.

■Figure 5

What is the role of testing in DevOps? In fact, it is more to play the role of containment and coordination. To give a simple example, our software is going to be released online. At this time, the developer should submit the code, and then the QA test is correct, and then the product is allowed to be handed over to the Ops for deployment. If there is a problem online, Ops will first feedback to QA through the monitoring platform for verification, and then feedback to the development after the verification is passed, not directly to the development. So in this process it is more of a containment and coordination role.

DevOps, where Dev development corresponds to CI (continuous integration) in CI/CD tools, and Ops corresponds to CD (continuous deployment). For testing, the counterpart is CT (continuous testing). The concept of continuous testing is introduced in the upper left corner of the figure.

02 Practice

1. GitLab Continuous Testing Best Practices

The main idea expressed in Figure 6 is to integrate software testing into the entire software development workflow, which is divided into two important parts: the first is to use GitLab's CI/CD (or other platforms) to Automated testing tools are integrated. The purpose of integrating CI/CD and automated testing tools is to achieve full automation. In this way, after the developer code is submitted or the test script is updated, a comprehensive test can be triggered immediately through CI/CD to completely solve the efficiency problem. CI/CD integrates automated testing tools, which is similar to the concept of space station we call. Extreme Fox GitLab or the DevOps platform used by various companies is essentially a core warehouse, and testing tools can be connected to it in a modular way. Finally spliced into a complete space station.

■Figure 6

The second important content is the test left shift, the test right shift and the quality access control. The test left shift is the test pre-test. The testers participate in the review of the requirements to solve the problem of everyone's understanding of the requirements. The test shift to the right is that after the program is released online, it needs to be monitored by Ops, and an alarm will be issued after encountering a problem. Both parts are important, but neither is as important as the intermediate code review part, since many companies are creating so-called quality gates. There will be two problems here: the first problem is how to manage the reports of so many test tools. Many teams directly send the reports to emails or chat tools, but how do the codes and versions match the reports? Another problem is the timing problem, which is how to associate the test with the code review. If the review passes, the code is merged into the main branch, but the test report does not pass. . Testing access control loses its meaning and value.

2. Extreme Fox GitLab software test base

For GitLab, the problem we want to solve is to combine testing with code review. Traditional test reports are discrete. Now we need to sort out these data and integrate all test tool reports into code review. , so that you can see these reports clearly for the first time during the review, and make sure that it is the report associated with the code mentioned this time. Code reviewers only need to sign off after referring to these reports.

For GitLab, we have always believed that we must aggregate these data uniformly for code review, which is very important for implementing code access control and code quality.

3. Preparation

Next, I will introduce how to carry out testing work within the enterprise, especially the start-up enterprise, when there is nothing? As shown in Figure 7, the first is about the preconditions. Here I listed three points. The most important thing is that you must have version control first, which means that you have a platform for code management, which is the premise of testing. Then there is CI/CD, which mainly solves the problem of efficiency, and it can be used as the integration platform, that is, the core cabin, to connect with the testing tools. The third point is that if there are no professional testers in the enterprise or the scale has not been completed, if we still want to do some basic tests (most of the simple tests are mainly manual), then we can use code scanning Tools, such as code quality scanning and code security scanning, can make up for the shortcomings of manual testing to a certain extent, such as variable name naming and incomplete comments.

■Figure 7

After meeting these conditions, if you have the ability to carry out automated testing, combined with the test pyramid theory just mentioned, it is generally believed that the bottom is the most important, normally unit testing should be the most important, but in fact here I rank unit testing in In the second place, put the interface test in the front. This is because for domestic enterprises and developers, there is generally no culture of unit testing, and most developers are in a state of work saturation. If they write unit tests at this time, it will cause greater resistance. . For interface testing, you can use some visual tools, which can implement automated testing of interfaces without programming capabilities.

4. Interface test

Figure 8 shows the advantages of interface testing. The most important points are points 4 and 5. The fourth point is that interface testing can be carried out through visual tools. At this time, the threshold is relatively low, and it is easier for testers to grasp. No need to add too much burden. The fifth point is that the interface will be more stable than the UI level, so its value can be more reflected after you automate it. In terms of process, testers generally use these visual tools to write test cases, then export them as test scripts, and then use the UI-free implementations of these tools (such as Docker containers) to load the scripts into them, so as to realize automated interface testing. These two points are relatively easy to implement at the practical level (landing level), so interface testing should be the first to be implemented for start-up companies.

■Figure 8

How to do interface testing is not so specific here. I briefly summarize it below the figure. There are many tools involved in interface testing, such as interface document management, interface testing tools, mock tools, performance testing tools, etc. For start-ups, I recommend everyone to use integrated tools. In fact, there are many integrated tools in China. They can help us manage interface documents, do interface testing, and even do performance testing. For start-ups, it simplifies work, which in turn increases efficiency.

5. Performance test

The performance test can be carried out next, because most of the performance tests are almost based on the interface, so after the interface test is completed, the performance test can be carried out incidentally. The types of performance tests are shown in Figure 9, such as load, pressure, immersion, and shock tests. Before conducting performance testing, it is important to know exactly what type of performance testing you are going to do. Performance testing is not a strong requirement, but a non-functional requirement. Different types have different purposes, so it must be done with a clear purpose. In the middle of Figure 9 is the core process of performance testing given by Microsoft, you can refer to it to do it.

■Figure 9

6. Unit testing

Unit testing can be done below. In fact, it can also be done in advance to the first stage. I did not put it at the top because the domestic unit testing is divided into two factions. Some people support it, and some people oppose it. , For enterprises, as long as they pay attention to the quality of products and the ability of personnel is satisfied, they can try to do it first. It can be considered from three aspects, as shown in Figure 10, that is, "who did it, how to measure it, and the effect".

■Figure 10

There is no doubt about "who does it". Unit testing is not done by testers. It must be done by developers themselves. Agile testing is carried out abroad, such as pair programming and pair development mentioned in Extreme Programming. Everyone does unit testing with each other. But it is more difficult to implement in the country. "Testing" is to verify whether the variables, conditions, paths, and resources of a single program or a single function method are correct. In fact, in the normal development process, developers will do self-tests. This self-test may be the steps that developers pack themselves and call from the API level or UI level to the written program methods (call from the outside to the inside) , to see if it is correct, which is relatively inefficient. It may also be that the developer writes some vest programs to verify the correctness of the function. In a sense, this is the work of unit testing, but it is not professional enough. You might as well write it according to the formal process of unit testing. "Effect" mainly includes two aspects. Direct indicators include unit test pass rate, coverage rate, and use case status. Indirect indicators are the overall trend of bugs after unit testing and whether the bug rate is reduced. Domestic enterprises attach great importance to indicators, even to the point of a deformity, so let's make a special statement. If you want to do unit testing, do not take the coverage of unit testing as the most important indicator, or require that the coverage of unit testing must reach 80% or 90%, which is absolutely not acceptable. In doing so, unit testing becomes a process and a formality. In addition, the purpose of unit testing is to verify the correctness of functional modules. Especially for complex function verification, there is no need to cover very simple functions, so do not pursue high coverage.

7. UI testing

The last is the UI test, which is divided into several platforms as shown in Figure 11. From the platform point of view, it is divided into App platform, web (such as the web platform of BS architecture), desktop platform (including not only PC, hardware terminal, host computer) The computer actually belongs to the desktop side). In addition, UI testing is mainly divided into two categories in terms of categories. One is logic, which refers to whether the sequence of operations and functions on the interface are correct. In this category, it is suggested that you must automate more than manual work, because many traditional enterprises rely on manual work when they do not develop automated testing tools, so automation can give full play to its advantages and replace a large number of repetitive operations. Another category is vision. Many companies will look at fonts, font sizes, copywriting, effects, etc. when verifying such UIs. Instead, I think this part is best done manually instead of automating it with great fanfare, because the cost of automation is high and the benefits are relatively high. Low. I don't think it's necessary to do OCR, pick text, identify, and judge copywriting, just glance at it. Putting UI testing at the end is also because of its conditions. Here I list ten conditions, which are also widely recognized in the industry. If these ten conditions are met, UI testing can be carried out. If these ten conditions are not If you are particularly satisfied, there is no need to do it.

■Figure 11

In terms of this process, UI testing is not much different from other tests. There are two modes when running: the first is to run in GUI mode, which is to do automated testing in UI mode. In this case, a special machine is required. For example, if there is a host computer, the host computer will be moved with the hardware as a special machine for testing, and then the automated test script will run on this special machine. Other operations, because the GUI is exclusive. Most automated UI testing for apps and desktop platforms is likely to be done this way.

Another mode is headless, which is to run software tests in a UI-less mode, which is mainly used in web UI tests, such as the ability based on the Chrome browser. Like selenium, it can implement a headless UI-free mode, running in a container, the advantage is that it can be concurrent, and the efficiency and performance are much higher than dedicated machines. The UI testing process is similar to the interface testing and performance testing just mentioned.

03 Q&A session

1. How do startups configure a dedicated testing team?

First of all, let’s look at the product. In the initial incubation stage, there are basically no professional testers or part-time developers. The small team’s testing is basically part-time. When the project makes some appearances, it can be used as a long-term development project. , you can build teams and keep staffing. For start-up teams I have experienced, in most cases, the initial ratio of development testers is 4:1 or 5:1, and the number of testers is relatively insufficient. Under such circumstances, it is not suitable to do the work on tools and processes with great fanfare. When the number of people in the entire team exceeds ten people or the product is very stable, and then add testers, then agile testing and automated testing can be carried out. Some companies may also introduce some outsourced resources in order to reduce costs, that is, outsourced personnel to do manual testing, and then core personnel to do automation, and these core personnel slowly turn to the direction of test development.

2. When the logic changes frequently, the maintenance cost is very high. How to automate it at this time?

If the logic changes frequently, it is recommended not to do automation anymore, because the changes are too fast, and it is appropriate to add automation after stabilization. I think it is more appropriate to use the pure manual method in the frequently changing situation, because as a start-up enterprise, everything is still based on simplicity and efficiency, and there is no need to pursue perfection and standards, which is not a good thing for enterprises.

3. When is the appropriate time to start a separate performance test?

At the beginning of product design, the performance of the product is actually an indicator of the product. If the product is aimed at the enterprise and is used by dozens of people, performance testing is not so important in a sense. And e-commerce platforms, such as 2C products, are used by thousands or even hundreds of thousands of people, and they must carry performance indicators. In this case, the sooner the better, it can be carried out in stages. Don’t wait for the launch before the launch. If it is launched before the launch, once the problem is found, the cost of solving the performance problem is very large. If the agile method is adopted, one iteration every two weeks, then the test will be conducted once every two weeks to solve the problem in advance. If it is a traditional development model, such as a six-month project, then at least one test will be conducted every month.

Upcoming Events

On the afternoon of July 16th, the 4th session of Shengwang Developer Entrepreneurship Lecture will be titled " How can the entrepreneurial team ensure the safety and compliance of product business? " Everyone brings wonderful sharing.

It's better to take action, scan the QR code or click here to register!