How to build an Apache APISIX plugin from 0 to 1 for back-end novice

In the past few months, community users have added many plugins to Apache APISIX, enriching the ecosystem of Apache APISIX. From the user's point of view, the emergence of more diverse plug-ins is undoubtedly a good thing. On the basis of improving the high performance and low latency of Apache APISIX, they meet the users' more expectations for the gateway, that is, "one Station" and "Multifunction".

How do community contributors develop plugins for Apache APISIX? None of the articles on the Apache APISIX blog seem to detail the process of developing plugins. So this time, let's change our perspective, from the perspective of plugin developers, let's take a look at the whole process of the birth of a plugin!

This article records the of developing the file-logger plugin by a front-end engineer no back-end experience. Before explaining the implementation process in detail, let me briefly introduce the functions of file-logger .

Features

file-logger supports the use of Apache APISIX plug-in metadata to generate custom log formats. Users can use the file-logger plug-in to append JSON-formatted request and response data to log files, or push the Log data stream to a specified location.

Imagine that when monitoring the access log of a route, we often focus not only on the value of some request and response data, but also want to write the log data to the specified file separately. At this time, you can use the file-logger plug-in to help achieve these requirements.

Pasting blocks outside Docs is not supported

In the specific implementation process, we can use file-logger to write the log data to the specified log file separately to simplify the process of monitoring and debugging.

development implementation process

After introducing the functions of file-logger , everyone has a better understanding of this plugin. The following is a detailed explanation for you. , who has no server-side experience, , how I completed the plug-in for Apache APISIX from 0 and added corresponding tests.

Determine plugin name and priority

Open Apache APISIX Plug-in Development Guide , and determine the following items in order:

1. Determine the plugin category.
2. Prioritize plugins and update conf/config-default.yaml file.

Because the file-logger developed this time is a log type plug-in, I refer to the name and order of the existing log plug-ins in Apache APISIX, and put file-logger here:

After consulting other plugin authors and enthusiastic members of the community, the plugin's name file-logger and priority 399 were finally confirmed.

It should be noted that the priority of the plugin is related to the execution order. The higher the priority value, the higher the execution. The ordering of plugin names is independent of execution order.

Create a minimal executable plugin file

After confirming the plugin name and priority, we can create our plugin code file in the apisix/plugins/ directory. There are two points to note here:

1. If the plugin code file is created directly in the apisix/plugins/ directory, there is no need to change Makefile file.
2. If your plugin has its own code directory, you need to update Makefile file. For detailed steps, please refer to Apache APISIX Plugin Development Guide .
3. Next we create the file-logger.lua file in the apisix/plugins/ directory.
4. Then use the official example of example-plugin as a reference to complete an initialized version.

-- 在头部引入我们所需要的模块
local core         =   require("apisix.core")
local plugin       =   require("apisix.plugin")
local ngx          =   ngx

-- 声明插件名称
local plugin_name = "file-logger"

-- 定义插件 schema 格式
local schema = {
    type = "object",
    properties = {
        path = {
            type = "string"
        },
    },
    required = {"path"}
}

-- 插件元数据 schema
local metadata_schema = {
    type = "object",
    properties = {
        log_format = log_util.metadata_schema_log_format
    }
}


local _M = {
    version = 0.1,
    priority = 399,
    name = plugin_name,
    schema = schema,
    metadata_schema = metadata_schema
}

-- 检查插件配置是否正确
function _M.check_schema(conf, schema_type)
    if schema_type == core.schema.TYPE_METADATA then
        return core.schema.check(metadata_schema, conf)
    end
    return core.schema.check(schema, conf)
end

-- 日志阶段
function _M.log(conf, ctx)
    core.log.warn("conf: ", core.json.encode(conf))
    core.log.warn("ctx: ", core.json.encode(ctx, true))
end


return _M

After completing a minimum available plug-in file through the example of example-plugin , you can output the plug-in configuration data and request-related data information to the error.log file through core.log.warn(core.json.encode(conf)) and core.log.warn("ctx: ", core.json.encode(ctx, true)) .

Enable the plugin and test

Next, create a test route to test whether the plug-in can successfully print the plug-in data and request-related data information we configured for it to the error log file.

1. Prepare a test upstream locally (the test upstream used in this article is 127.0.0.1:3030/api/hello that I created locally).
2. Create a route with the curl command and enable our new plugin.

curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
  "plugins": {
    "file-logger": {
      "path": "logs/file.log"
    }
  },
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "127.0.0.1:3030": 1
    }
  },
  "uri": "/api/hello"
}'

Then you will see a status code of 200 , indicating that the route has been successfully created.

3. Run curl command to send a request to this route to test whether the file-logeer plugin has been started.

$ curl -i http://127.0.0.1:9080/api/hello
HTTP/1.1 200 OK
...
hello, world

4. In the logs/error.log file there will be a record like this:

It can be seen that the path: logs/file.log we configured for the plugin in the conf parameter has been successfully saved. So far we have successfully created a minimal usable plugin that prints the data for the conf and ctx parameters in the log phase.

After that, we can write core functions for it directly in the file-logger.lua plugin code file. Here we can directly run apisix reload command to reload the latest plugin code without restarting Apache APISIX.

Write core functionality for the file-logger plugin

The main function of the file-logger plugin is to write log data. After inquiring and consulting materials, I learned about Lua's IO library: https://www.tutorialspoint.com/lua/lua_file_io.htm , so I confirmed that the functional logic of the plugin is roughly as follows:

1. After each request is accepted, the log data is output to the plugin configured

   path

   go in.

   1. First, in the log phase, get the value of file-logger in path through conf .
   2. Then, the operations of file creation, opening, writing, flushing the cache, and closing are completed through the Lua IO library.
2. Handle errors such as failure to open a file, failure to create a file, etc.

local function write_file_data(conf, log_message)
    local msg, err = core.json.encode(log_message)
    if err then
        return core.log.error("message json serialization failed, error info : ", err)
    end

    local file, err = io_open(conf.path, 'a+')

    if not file then
        core.log.error("failed to open file: ", conf.path, ", error info: ", err)
    else
        local ok, err = file:write(msg, '\n')
        if not ok then
            core.log.error("failed to write file: ", conf.path, ", error info: ", err)
        else
            file:flush()
        end
        file:close()
    end
end

3. Referring to the source code of the http-logger plug-in, the method of transferring the log data to the writing log data and some judgment and processing of the metadata have been completed.

function _M.log(conf, ctx)
    local metadata = plugin.plugin_metadata(plugin_name)
    local entry

    if metadata and metadata.value.log_format
        and core.table.nkeys(metadata.value.log_format) > 0
    then
        entry = log_util.get_custom_format_log(ctx, metadata.value.log_format)
    else
        entry = log_util.get_full_log(ngx, conf)
    end

    write_file_data(conf, entry)
end

Validate and add tests

Verify collection logging

Because the file-logger plugin has been enabled when creating the test route, and path is configured as logs/file.log , we only need to send a request to the test route to verify the results of log collection:

curl -i http://127.0.0.1:9080/api/hello

In the corresponding logs/file.log , we can see that each record is saved in JSON format. After formatting one of the data as follows:

{
    "server":{
        "hostname":"....",
        "version":"2.11.0"
    },
    "client_ip":"127.0.0.1",
    "upstream":"127.0.0.1:3030",
    "route_id":"1",
    "start_time":1641285122961,
    "latency":13.999938964844,
    "response":{
        "status":200,
        "size":252,
        "headers":{
            "server":"APISIX\/2.11.0",
            "content-type":"application\/json; charset=utf-8",
            "date":"Tue, 04 Jan 2022 08:32:02 GMT",
            "vary":"Accept-Encoding",
            "content-length":"19",
            "connection":"close",
            "etag":"\"13-5j0ZZR0tI549fSRsYxl8c9vAU78\""
        }
    },
    "service_id":"",
    "request":{
        "querystring":{

        },
        "size":87,
        "method":"GET",
        "headers":{
            "host":"127.0.0.1:9080",
            "accept":"*\/*",
            "user-agent":"curl\/7.77.0"
        },
        "url":"http:\/\/127.0.0.1:9080\/api\/hello",
        "uri":"\/api\/hello"
    }
}

At this point, the process of verifying the collection of log records is over. The verification result indicates that the plug-in successfully started and returned the expected data.

Add tests for plugins

As for the code in the add_block_preprocessor part, I was confused when I first started writing it because I didn't learn Perl myself. It was only after asking about the correct way to use it: if we did not write the relevant request assertion and no_error_log assertion in the data section, then the default assertion is as follows:

--- request
GET /t
--- no_error_log
[error]

After comprehensively referring to some other log test files, I created the file-logger.t file in the t/plugin/ directory.

Each test file is divided into a preamble and a data part by __DATA__ . Since the official website test related documents are not clearly archived and classified, for more specific content, please refer to the relevant information at the end of the article. Below I list one of the test cases completed after referring to the relevant materials:

use t::APISIX 'no_plan';

no_long_string();
no_root_location();

add_block_preprocessor(sub {
    my ($block) = @_;

    if (! $block->request) {
        $block->set_value("request", "GET /t");
    }

    if (! $block->no_error_log && ! $block->error_log) {
        $block->set_value("no_error_log", "[error]");
    }
});


run_tests;

__DATA__

=== TEST 1: sanity
--- config
    location /t {
        content_by_lua_block {
            local configs = {
                -- full configuration
                {
                    path = "file.log"
                },
                -- property "path" is required
                {
                    path = nil
                }
            }

            local plugin = require("apisix.plugins.file-logger")

            for i = 1, #configs do
                ok, err = plugin.check_schema(configs[i])
                if err then
                    ngx.say(err)
                else
                    ngx.say("done")
                end
            end
        }
    }
--- response_body_like
done
property "path" is required

At this point, the plug-in adding test link is over.

Summarize

The above is the whole process of implementing an Apache APISIX plugin from 0 to me as a back-end novice. I did encounter a lot of pitfalls in the process of developing the plug-in. Fortunately, there are many enthusiastic bigwigs in the Apache APISIX community to help me solve my doubts, which made the development and testing of the file-logger plug-in smooth. If you are interested in this plugin, or want to see the plugin details, you can refer to Apache APISIX official document .

At present, Apache Discussion APISIX is also developing other plugins to support the integration of more services. If you are interested in this, please feel free to initiate a discussion on GitHub , or communicate through the mailing list .