Nebula integration test framework reconstruction based on BDD theory (part 1)

This article was first published in the Nebula Graph public NebulaGraphCommunity , Follow to see the technical practice of the large factory graph database.

The evolution of the testing framework

So far, during the development of Nebula Graph, the test framework has undergone three major changes, as shown in the following figure. In the continuous evolution, the team has accumulated some experience and lessons. I hope to use this article to make a simple introduction and sorting out.

For a database product, the importance of testing is self-evident, and it cannot be overemphasized. Therefore, no matter who the testing framework is switched to, there is always only one starting point: convenient and fast accumulation of test cases to ensure the stability of Nebula Graph function. The "convenience and fast" mentioned here is not limited to the "developers" group, but all users who need to face Nebula Graph, which may be operations, documentation, or even non-technical related personnel. In order to achieve this goal, it is best to allow users to perform "codeless programming" without even programming.

Throughout most of the database products, often set their own custom text rules, developers to submit test based on this set of rules, we have had preliminary this attempt , taking into account the follow-up to scratch too many customized features, In addition, the user needs to learn a new set of rules, and ultimately there is no real switch over. Until we started to do the openCypher-compatible MATCH function, we noticed that TCK this repo , although this is a compatibility test suite, but it provides us with a new idea for the integration test of Nebula Graph. One reason why the aforementioned attempts are not easy to implement is that the result set returned by Nebula Graph is a composite data structure that may contain points, edges, and paths. It is not impossible to use a JSON-like approach, but it is not elegant and concise. When there are more result sets, there is a suspicion that the "form" is greater than the "content", and the description of the structure is far more than the data you really care about, which is verbose and annoying. The set of description rules for describing points, edges, and paths formulated in TCK are simple and intuitive enough, and fit the Pattern statement in MATCH, echoing before and after, as long as users who have used openCypher, it is easy to accept and understand. It is only for the strong schema requirements of Nebula Graph, and some expansion of its rules is needed, but it does not hurt. In view of the above advantages, let us firmly move towards the BDD test framework.

The end-to-end functional test of Nebula Graph is actually a "black box" test. The main thing to accomplish is abstracted out: execute an nGQL statement and compare the returned result set.

First, by comparing the complexity of the following test cases, we can intuitively feel the progress of each time, from top to bottom: 1. GTest-based testing; 2. pytest-based testing; 3. BDD-based testing.

[GTest-based test]

[Test based on pytest]

[BDD-based test]

It can be seen from the above comparison that we are getting closer and closer to the truth of "testing". As long as we care about input and output, we don't need to code and assemble test data, and supplemented with some small automation tools, it greatly reduces the threshold for adding use cases. .

Expectation and realization

Before expanding the TCK-based testing framework, we set the following expected goals for this upgrade:

1. It is easy to add use cases, and it is convenient to construct expected data;
2. Support importing other test data sets;
3. Reuse the flexibility of the pytest framework, especially mechanisms such as plugins and fixtures;
4. Compatible with Match TCK use cases;

In order to achieve the above goals, we started a new technology selection and module design. Before building Nebula Graph's own TCK test framework, the first choice is to choose a "appropriate" test framework. The basic requirements of this framework are as follows:

1. Complete support for BDD-based testing;
2. Convenient, flexible and expandable;
3. It is best to be compatible with existing pytest use cases.

There are many test frameworks that implement BDD, even in the python language environment, there is a multiple choice question , such as pytest-bdd , behave and so on. In view of the third point in the above goals, we chose to build the entire testing process of Nebula Graph based on pytest-bdd. pytest-bdd is a plug-in of pytest, which can well support the features of BDD and can directly use the functions of pytest, which is more in line with our expectations.

After selecting each module testing framework, we began to design the entire testing process, the general structure can be divided into five parts: the ConnectionManager , DataLoader , Parser , Comparator , Reporter .

ConnectionManager

Manage the connection with nebula graph, including error retry, error filtering and other functions.

DataLoader

Read the CSV data file, analyze the data type in the configuration, and splice the INSERT statement that inserts the data, etc.

Parser

Parse the strings of points, edges and paths described in TCK and convert them into the Value structure defined by Nebula for easy comparison.

Comparator

Responsible for the value comparison of different Value structures, including basic data types and composite data types. Composite data types include: List, Map, Set, Vertex, Edge, Path, etc.

Reporter

Better output the customized functions such as the position and line number of the wrong nGQL statement in the feature file.

Modules are mutually independent and interrelated. Together with the different scopes of fixtures in pytest, the isolation and testing of different scenarios can be completed well.

What is BDD

Previously mentioned many times BDD, we understand TDD and DDD are more likely to what is BDD also holds a doubt . The so-called BDD is actually a testing method evolved from TDD, that is, behavior-driven development. The test is completed by writing test cases in natural language, which is more friendly to participants other than developers, thus narrowing the distance between developers and users. In our practice, we found that, in fact, this set of methods of BDD is not only effective for managing software quality, but also a good supplement to complex demand management. The user’s requirement description is no longer limited to the complex scenario description, and the functional requirements can be aligned with the developer through the desired query statement, process, and conclusion. These requirement files turn into test scenario use cases after the function is developed. Kill two birds with one stone.

Speaking of BDD, Gherkin language is inseparable. It defines a set of basic grammatical rules to effectively organize the structure of ordinary text so that BDD testing tools can understand the content described in the text. The file storing the Gherkin language text uses .feature as the extension, which can describe many scenarios (Scenario) and the steps in each scenario (Given/When/Then). These grammatical rules are very simple and easy to understand, and the number of keywords is small, so reading Gherkin's test text is like a "question and answer" dialogue, which is easy to learn.

The testing framework of Nebula Graph expects to use the BDD method to create a pure "black box" testing process. Regardless of whether the user is a developer or not, only two points need to be paid attention to. What is the input nGQL and what is the expected return data? Only in this way can the mental burden of users adding use cases be reduced, and it is convenient for them to contribute to the Nebula Graph. Within half a year after we completed the framework transformation, we had accumulated about 2500 test cases internally, which provided strong quality assurance for the reconstruction of the 2.0 project. All use cases are divided into categories and placed in the tests/tck/features directory in the repo. These use cases are essentially a guide to the use of nGQL. The next time the user encounters a tricky problem and does not know how to describe it in nGQL, You can refer to the use case here first.

to sum up

This article briefly reviews the evolution of the Nebula Graph test framework, and will show you the functions that the current test framework has completed and how to use it to test changes to the Nebula Graph source code.

Exchange graph database technology? Please join Nebula exchange group under Nebula fill in your card , Nebula assistant will pull you into the group ~