Test cases must not be casual, record the thinking caused by an abnormal test case

one

Do you usually write test cases?

I used to write test cases only for business interfaces, one for each interface, and only one data case for testing. You can run through it. If you want different scenarios, change the data. run again. Simple and easy.

But since I started maintaining open source in my spare time, my understanding of test cases has started to deepen. Even I have now elevated the status of test cases to the same important position as core code. I once joked that writing core code without writing test case code is hooliganism.

Open source projects are facing everyone. Everyone has different environments and different project structures. The versions of jdk, spring system, and third-party dependencies are different. So open source frameworks must work well in all scenarios. With so many function points and so many scenarios, even if I am the author, it is impossible to remember so many details just by familiarity. At this time, test cases are very important, and they are the most critical quality assurance of the entire project. A lot of times, I rely on test cases to find some small-edge bugs. At present, my open source project has 870 test cases, covering more than 90% of the scenarios.

This article explores a problem with the mechanism of test case execution caused by test cases.

two

The cause of the incident was that a small partner in the group found that when a certain unit test case had an incorrect configuration item, the spring context was executed twice, but in the case of correct configuration, it was only started normally once. This puzzled him, thinking that there was a problem with the framework.

The reason why he thinks that spring has started twice is that he sees that the logo print of springboot appears twice in the log, and the two times the same error is reported:

 .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.0.5.RELEASE)
 
com.yomahub.liteflow.exception.ELParseException: 程序错误，不满足语法规范，没有匹配到合适的语法,最大匹配致[0:7]
    at com.yomahub.liteflow.builder.el.LiteFlowChainELBuilder.setEL(LiteFlowChainELBuilder.java:124) ~[liteflow-core-2.8.2.jar:na]
    at com.yomahub.liteflow.parser.helper.ParserHelper.parseOneChainEl(ParserHelper.java:391) ~[liteflow-core-2.8.2.jar:na]
    at com.yomahub.liteflow.parser.el.XmlFlowELParser.parseOneChain(XmlFlowELParser.java:20) ~[liteflow-core-2.8.2.jar:na]
    at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_292]
    at com.yomahub.liteflow.parser.helper.ParserHelper.parseDocument(ParserHelper.java:217) ~[liteflow-core-2.8.2.jar:na]
    at com.yomahub.liteflow.parser.base.BaseXmlFlowParser.parse(BaseXmlFlowParser.java:40) ~[liteflow-core-2.8.2.jar:na]
      .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.0.5.RELEASE)
 
com.yomahub.liteflow.exception.ELParseException: 程序错误，不满足语法规范，没有匹配到合适的语法,最大匹配致[0:7]
    at com.yomahub.liteflow.builder.el.LiteFlowChainELBuilder.setEL(LiteFlowChainELBuilder.java:124) ~[liteflow-core-2.8.2.jar:na]
    at com.yomahub.liteflow.parser.helper.ParserHelper.parseOneChainEl(ParserHelper.java:391) ~[liteflow-core-2.8.2.jar:na]
    at com.yomahub.liteflow.parser.el.XmlFlowELParser.parseOneChain(XmlFlowELParser.java:20) ~[liteflow-core-2.8.2.jar:na]
    at java.util.ArrayList.forEach(ArrayList.java:1259) ~[na:1.8.0_292]
    at com.yomahub.liteflow.parser.helper.ParserHelper.parseDocument(ParserHelper.java:217) ~[liteflow-core-2.8.2.jar:na]
    at com.yomahub.liteflow.parser.base.BaseXmlFlowParser.parse(BaseXmlFlowParser.java:40) ~[liteflow-core-2.8.2.jar:na]

The test case code is:

 @RunWith(SpringRunner.class)
@TestPropertySource(value = "classpath:/whenTimeOut/application1.properties")
@SpringBootTest(classes = WhenTimeOutELSpringbootTestCase.class)
@EnableAutoConfiguration
@ComponentScan({"com.yomahub.liteflow.test.whenTimeOut.cmp"})
public class WhenTimeOutELSpringbootTestCase {

    @Resource
    private FlowExecutor flowExecutor;

    //其中b和c在when情况下超时，所以抛出了WhenTimeoutException这个错
    @Test
    public void testWhenTimeOut1() throws Exception{
        LiteflowResponse response = flowExecutor.execute2Resp("chain1", "arg");
        Assert.assertFalse(response.isSuccess());
        Assert.assertEquals(WhenTimeoutException.class, response.getCause().getClass());
    }
}

The open source framework is at the source code level, and it is impossible to actively start the spring context again (in fact, I don't know how to do it). And it is normal when configured correctly. And spring's @Configuration is also started twice. From the thread stack, it is also triggered by Junit:

It is worth mentioning that the error reported is in the springboot startup link. Therefore, it did not enter the test case code decorated by @Test at all. So it doesn't matter what the code is written. I tested it, if an exception is thrown in the test code, the spring context is only started once.

So this problem may end here, because it is not a problem of the framework itself. Junit itself triggers two actions to initialize spring when it fails to start spring, which may be a retry mechanism of Junit. This is not under my control. Anyway, if there is a real error, it will be thrown out. I don’t need to initialize it several times, and it will not affect the overall effect of my test case. Just change the specific test case to the right one.

three

But I suddenly thought about the Spring loading mechanism of the test case when I was dealing with a test case, so I thought of the previous problem. Suddenly realized.

The structure of our use cases is generally that a test case represents a large scene, and each method in it represents a specific case. Assuming that a class has 10 test specific use cases, when you click Run Test on the class, how many times will spring be initialized.

The answer is 1 time. In order to speed up the process of running test cases, it is impossible for springboot test to initialize spring for every method. The spring context in this class will be cached, and these 10 methods will share the same spring context.

The specific operation mechanism is: when you click on the Run Test of the class, it will initialize the spring first, and then start to run the test methods one by one. When the test method is running, if it is found that the spring has not been initialized, the spring will be initialized again. This explains that when we run the run test of the method separately, spring is also initialized.

Now we can explain the previous problem, because the initialization failed, and when the method was run, it was found that it had not been initialized, so it was initialized again.

But for different Test classes, it will still be initialized multiple times. That is to say, each class will initialize spring again. This should be found when you run multiple test cases.

Four

One more question: Has anyone encountered the problem that when running all the test cases, there will always be a few errors, but a single run is completely normal?

If you have encountered it, you must have ignored the following points:

If you choose to run all test cases, although spring is initialized once for each test case class, the JVM is only started once from beginning to end. And your static variables defined in the class will not change with spring startup. When you run all of them, it is possible that some of the static variables referenced by your faulty test case are still data left over from the previous test case. So there may be an error. When running a single operation, there is no such phenomenon.

If you encounter this situation, you have to use the @AfterClass annotation in the test case, and clear the static variables in the test case in the method declared by the annotation. This will work together. For example, each of my test cases inherits a BaseTest method and writes this method in it to clear the static cache:

 public class BaseTest {
    @AfterClass
    public static void cleanScanCache(){
        ComponentScanner.cleanCache();
        FlowBus.cleanCache();
        ExecutorHelper.loadInstance().clearExecutorServiceMap();
        SpiFactoryCleaner.clean();
        LiteflowConfigGetter.clean();
    }
}

five

About how to write test cases, what are the common ways to write them. I won't go into too much detail here, you should be able to find a lot of tutorials by Baidu, or if you are interested, you can also read the test cases in my open source project LiteFlow.

In addition to ensuring the quality of your project, test cases can also clearly see how many lines of code your entire test cases cover. The test cases I have here are written as separate projects. Used to distinguish the core project package.

Then configure the task of executing testcase separately in IDEA:

Then go to the run xxx with coverage button to run the test case:

Between multiple test projects, after running one, a dialog box will pop up asking if you want to add this result to the total result, just click add:

All your test case projects are running, and the following report page will appear on the right:

Here at the top you can see that the number of lines covered by my entire test case is 79%. But this does not mean that the project covers only 79% of the scene. Line coverage and functional scenario coverage are two concepts. Here, it only represents the proportion of all lines of code after all test cases are run.

Finally, I hope that everyone can't ignore the test case. Although I sometimes feel sick when I write it, in the end you will appreciate its sweetness.