Help developers to interpret APISIX test cases in an all-round way

Background Information

Apache APISIX is a cloud-native API gateway under the Apache Software Foundation. It has the characteristics of dynamic, real-time, and high performance. It provides load balancing, dynamic upstream, grayscale release (canary release), service fuse, identity authentication, Observability and other rich traffic management functions. You can use APISIX to handle traditional north-south traffic, as well as east-west traffic between services. At the same time, it also supports use as K8s Ingress Controller.

Under normal circumstances, to ensure the normal operation of the software, we generally use various technologies and methods to check the functions of the software manually or automatically to ensure that it is running normally before the software is launched. We call this operation QA (testing). Testing is generally divided into unit testing, E2E testing and chaos testing.

Unit testing is used to check the correctness of a single module (such as checking whether the serialization/deserialization of a certain RPC, data encryption and decryption are normal), but this test lacks a global perspective on the system. The E2E test (ie end-to-end test) can make up for the insufficiency of the unit test. The test runs the entire system and external dependent services, and checks the integration of the system and other systems through real software calls; chaos tests pass Create unexpected situations between various components of the system, such as OOM Kill, network interruption, etc., to test the tolerance and capability of errors in the entire system. The testing of APISIX is more inclined to E2E testing to ensure the correctness of its own functions and integration with other systems.

Introduction to APISIX Test Cases

As APISIX is the most active API gateway in the world, its stability and service robustness need to be guaranteed, so how to avoid potential errors in APISIX? This needs to be achieved through test cases.

A test script is not just a program file executed by the machine under test. For developers, the test script can be used to test all functions of the software, including the running status of the program under different configurations and different input parameters. For the user, the test provides a specific example of the use of a function module, such as: the configuration and input that the program can accept, and what kind of output results you want to get. If users encounter something they don't understand when referring to the documentation, they can refer to the existing test cases to find out whether there are similar usage scenarios.

In the APISIX project, Github Action is usually used to run CI tests and execute the test scripts shown in the figure below. Many APISIX developers encounter various problems when writing test cases. Hopefully, with this article, you can reduce the mistakes you make when writing APISIX test cases.

Write test cases

APISIX's test cases are written based on Test::NGINX test framework, which is a test environment implemented on the basis of Perl language, which can provide script-based automated testing capabilities, which is the current large-scale testing and quality assurance for APISIX work provides support. Of course, it doesn't matter if you don't use Perl, because you don't need to write Perl code in most scenarios, just use the ability of TEST::NGINX package, if you have special needs, you can combine it with Lua code to enhance .

The test cases of APISIX are stored in the ./apisix/t directory. Next, we will introduce how to write test cases by adding the opa plugin as an example.

1. You need to create a test file that ends with .t 7d6c4e315265a20e1015e16fd2ab797b---, eg ./t/plugin/opa.t . If you are adding features to existing functions, you can directly add test cases to the corresponding test files. And add the fixed format ASF 2.0 protocol in the file.
2. The main function of this part is to automatically add no_error_log opa.t so that there is no need to add error_log related code under each code Now, you can directly copy and use this code. In this way, some duplication of code can be reduced.

 add_block_preprocessor(sub {
    my ($block) = @_;

    if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
        $block->set_value("no_error_log", "[error]");
    }

    if (!defined $block->request) {
        $block->set_value("request", "GET /t");
    }
});

run_tests();

DATA

3. Each test case has a fixed beginning, and the general format is as follows.

 === TEST 1: sanity

=== The fixed syntax structure at the beginning of the test case, TEST ` `1 represents the first test case in this file. sanity is the name of the test. Usually named after the specific purpose of the test case.

4. Next is the body of the test case. Almost every plugin in APISIX will define some parameters and properties, and will pre-define JSON schema, so we need to check whether the input parameters of the plugin can be verified normally. In this way, we can check whether our input data is not It can be correctly verified by the rules of JSON schema.

 --- config
    location /t {
        content_by_lua_block {
            local test_cases = {
                {host = "http://127.0.0.1:8181", policy = "example/allow"},
                {host = "http://127.0.0.1:8181"},
                {host = 3233, policy = "example/allow"},
            }
            local plugin = require("apisix.plugins.opa")
            for _, case in ipairs(test_cases) do
                local ok, err = plugin.check_schema(case)
                ngx.say(ok and "done" or err)
            end
        }
    }
--- response_body
done
property "policy" is required
property "host" validation failed: wrong type: expected string, got number

By reading the source code of the opa plugin, you can see that opa plugin requirements host and policy must exist at the same time, so the definition of three a rule.

• Enter the correct parameters, including host and policy . So the return result will be done ;
• Enter only the host parameter. Does not meet the requirements of host and policy at the same time, so the expected return result of this test will be property "policy" is required ;
• The host value of the wrong type (integer) was entered. Because the host parameter must be set in the source code, the returned result will be property "host" validation failed: wrong type: expected string, got number .

 --- config
    location /t {
        content_by_lua_block {
...
                ngx.say(ok and "done" or err)
            end
        }
    }
--- response_body
done
...

In general, the function of /t needs to be used in each test case. For example, if you need to call Lua code and define location , you can use the method of content_by_lua_block Call some code to assist the test, and finally print the response information in the form of ngx.say , and then check whether the above program is running correctly by means of --- response_body . No need to manually enter request and error as we have added them automatically via script.

 local plugin = require("apisix.plugins.opa")
            for _, case in ipairs(test_cases) do
                local ok, err = plugin.check_schema(case)
                ngx.say(ok and "done" or err)

The above code represents the module that imports the APISIX Plugin opa plugin, and calls the plugin.check_schema function. Then pass the for loop to call the parameters and return the corresponding results according to the test situation.

5. Next, we need to configure an environment to use when testing. For plugins, it is to create a route, and then associate the plugin to the route. After the creation is complete, we can verify whether the internal logic of the plugin is implemented correctly by sending a request.

 === TEST 2: setup route with plugin
--- config
    location /t {
        content_by_lua_block {
            local t = require("lib.test_admin").test
            local code, body = t('/apisix/admin/routes/1',
                 ngx.HTTP_PUT,
                 [[{
                        "plugins": {
                            "opa": {
                                "host": "http://127.0.0.1:8181",
                                "policy": "example"
                            }
                        },
                        "upstream": {
                            "nodes": {
                                "127.0.0.1:1980": 1
                            },
                            "type": "roundrobin"
                        },
                        "uris": ["/hello", "/test"]
                }]]
                )
            if code >= 300 then
                ngx.status = code
            end
            ngx.say(body)
        }
    }
--- response_body
passed

在以上示例中， lib_test_admin c115f950a3720ff3b5ff5f56d3d5926d---导入到t函数， id 1 ，然后使用PUT 的方法, pass in these data. In this test, we did not check the data format, because the test case only needs to ensure that the Admin API can be used to create routes normally. Of course, we also need to judge the exception, if the status code is greater than or equal to 300 will print out specific information.

In APISIX's test cases, you will find that many of the tests include the following code:

 local t = require("lib.test_admin").test
local code, body = t('/apisix/admin/routes/1', ngx.HTTP_PUT)

The above code means to import lib.test_admin module and use the test function package to send the request, which reduces the repetitive code when we call HTTP interfaces such as APISIX Admin API, just simply call and check the returned result.

6. In the third test, we don't need to repeat the creation of the route because it was already created in the second test.

 === TEST 3: hit route (with correct request)
--- request
GET /hello?test=1234&user=none
--- more_headers
test-header: only-for-test
--- response_body
hello world

The above example defines request .因为我们在上一个测试中定义了---ed42ee869810dd0e4e908c87ac20c686 /hello / `test 的路径，所以我们可以通过 GET 的方法向 / `hello request, and then it will send a test=1234 and user=none query parameters. You can also add response headers by more_headers , such as adding a response header called test-header to requests sent to hello .

The above test is to test whether adding the correct request header can be successful. You can also add error headers in subsequent test examples and verify the results. POST请求，也可以把上述代码中的GET /hello POST /hello .

In some tests, you may need to create multiple upstreams or routes. In this case, you can define an array, then define these corresponding values in the array, and call for cyclically t function, and then let it call the Admin API interface of APISIX through put to create a route or upstream normally. This method is also a common method of testing, called: Table driving test, which is a method of driving tests through tables, which reduces some repetitive codes, such as opa2.t. For a detailed introduction, please click to read the original text and refer to the APISIX test case quick start video.

Run the test case

The test case requires the source code to install APISIX. Next, we will focus on how to run the test case and check for errors.

run locally

Typically, you can run the test case locally with the following command:

 PATH=/usr/local/openresty/nginx/sbin:/usr/bin PERL5LIB=.:$PERL5LIB FLUSH_ETCD=1 prove -Itest-nginx/lib -r t/admin

• PATH specifies the directory where openresty/nginx is located, which can avoid conflicts caused by incorrect configuration of some environments. If OpenResty in the environment is installed in other locations, it can also be specified by this command .
• PERL5LIB Specifies importing to local using Perl. Import the PERL library that exists in this path and some of the PERL libraries attached via environment variables.
• FLUSH_ETCD Specifies that after each test file is executed, all data will be cleared, it needs to call etcdctl function, you need to ensure that etcdctl executable file can be found in the PATH.
• prove Call the test program to start the test.
• -Itest-nginx/lib means to import Itest-nginx/lib this library.
• -r means to automatically find the test file. If a path is specified, all test files under this path will be searched.

The following is the normal execution result of the above command.

• t/admin represents the specified test case search path, which can also be specified to the only one .t file to limit.
  

If the test fails, the following message appears:

The above information will tell you which test case in which test file failed to execute.

Github run

In general, when submitting code in Github, the output is similar to testing locally.

First choose the wrong execution workflow, the main test cases are in the build series CI.

We can see that in this example, the line 416 an error. Through the error information, we can get that there is an error in a test case in a certain test file, and the developer can check and correct it. It should be noted that there may be some strange errors in CI, which may be caused by temporary exceptions in the CI environment. If the code in the corresponding module has not been modified, these errors can be ignored.

Summarize

This article mainly introduces the relevant process of testing, as well as the composition of APISIX test cases and how to write test cases. I hope that through this article, you can have a general understanding of APISIX test cases.

This article only mentions some core content in the APISIX test framework, and fails to cover all the content in the TEST::NGINX framework. In fact, there are many powerful capabilities in TEST::NGINX. We can pass Test::Nginx:: Socket 's documentation for more usage. If you want to learn more about writing test cases, check out the APISIX Test Cases Quick Start video.