Flaky test management in Cypress

Hover over the mouse and see the prompt message: This test both passed and failed when retried within a run

What is Flaky test? Cypress official website has the following definitions:

A test is considered unstable when it can pass and fail in multiple retry attempts without any code changes.

For example, a test is executed and fails, then the test is executed again without any changes to the code, but this time it passes.

When previously tested code fails, it's a strong signal that there is a new bug in the code. Before, the test passed and the code was correct; now the test fails and the code doesn't work. The goal of a good test suite is to make this signal as clear and directed as possible.

However, the flaky test is different. Flaky tests are tests that show pass and fail results using the same code. Given this, a failed test may or may not mean a new problem. And try to reproduce the failure by rerunning the test with the same version of the code, which may or may not cause the test to pass. We started to think these tests were unreliable, and eventually they lost their value. If the root cause is uncertainty in production code, ignoring tests means ignoring production bugs.

A statistic for your reference:

Google runs about 4.2 million tests on its continuous integration system. Of these, about 63,000 experienced erratic operation over the course of a week. While this accounts for less than 2% of Google's tests, it's still a big drag on Google engineers.

The above data comes from this blog .

Test retries are the basic mechanism that enables Cypress Dashboard to detect unstable tests. Therefore, test retries need to be enabled to take advantage of any erratic test management features provided by the cypress dashboard.

Test runs with unstable tests will be marked with the number of unstable tests in the Dashboard "Latest Runs" page. Flaky test runs can also be filtered in and out via the "Flaky" filter on this page.

As shown below:

https://github.com/SAP/spartacus/pull/14401

Click view details, and then select a specific cypress job view the details:

Any failure of multiple test run attempts triggered by test retries will result in a given test case being marked as unstable.

The first attempt failed, the second succeeded:

about flaky test in Spartacus.

Protractor is another Angular end-to-end testing framework besides Cypress:

For Spartacus UI, almost every part of the page is created dynamically (we need to make a backend call first).

Does the underlying Protractor call Selenium?

article how to write more stable tests with Selenium is worth reading.

More Jerry's original articles, all in: "Wang Zixi":