Apache APISIX integrates Kafka for efficient real-time log monitoring

Apache Kafka is an open source stream processing platform managed by Apache, written in Scala and Java, and provides unified, high-throughput, low-latency features for processing real-time data.

Its persistence layer is essentially a "large-scale publish/subscribe message queue following a distributed transaction log architecture", which makes it very valuable as an enterprise-grade infrastructure for processing streaming data. It is currently used by thousands of companies in areas such as high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Implementation: kafka-logger

Apache APISIX has provided kafka-logger plug-in support as early as version 1.2, and has been enhanced many times since then, and now it has very mature and complete functions. Supports pushing API request logs, and even request body and response body to Kafka cluster in JSON format.

When using kafka-logger , users can send a variety of data and customize the sent log format, and also support functions such as batching and sending logs or automatic retry.

how to use

Step 1: Start the Kafka cluster

The example in this article only demonstrates one startup method. For details of other startup methods, please refer to official document .

# 使用 docker-compose 启动一个具有 1个 zookeeper 节点、3个 kafka 节点的集群
# 同时还启动一个 EFAK 用于数据监控。
version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.2.1
    hostname: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: zookeeper:2888:3888

  kafka1:
    image: confluentinc/cp-kafka:6.2.1
    hostname: kafka1
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
    depends_on:
      - zookeeper

  kafka2:
    image: confluentinc/cp-kafka:6.2.1
    hostname: kafka2
    ports:
      - "9093:9093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka2:19093,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_BROKER_ID: 2
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
    depends_on:
      - zookeeper


  kafka3:
    image: confluentinc/cp-kafka:6.2.1
    hostname: kafka3
    ports:
      - "9094:9094"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka3:19094,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
      KAFKA_BROKER_ID: 3
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
    depends_on:
      - zookeeper

  efak:
    image: nickzurich/kafka-eagle:2.0.9
    hostname: efak
    ports:
      - "8048:8048"
    depends_on:
      - kafka1
      - kafka2
      - kafka3

Step 2: Create Topic

Next we create test Topic for log collection.

Step 3: Create a route and enable the plugin

The following commands can be used to create routes and enable the kafka-logger plugin.

curl -XPUT 'http://127.0.0.1:9080/apisix/admin/routes/r1' \
--header 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
--header 'Content-Type: application/json' \
--data-raw '{
    "uri": "/*",
    "plugins": {
        "kafka-logger": {
            "batch_max_size": 1,
            "broker_list": {
                "127.0.0.1": 9092
            },
            "disable": false,
            "kafka_topic": "test",
            "producer_type": "sync"
        }
    },
    "upstream": {
        "nodes": {
            "httpbin.org:80": 1
        },
        "type": "roundrobin"
    }
}'

The above code configures the kafka broker address, the target topic, the synchronized production mode, and the maximum number of logs contained in each batch. Here we can first set batch_max_size to 1, that is, write a message to Kafka every time a log is generated.

With the above settings, the function of sending API request logs under the path /* to Kafka can be realized.

Step 4: Send the request

Next we send some requests through the API and record the number of requests.

# 向 API 发送 10 次请求
curl http://127.0.0.1:9080/get

As you can see from the image below, some log messages have been written to the test topic we created. Click to view the log content, you can find that the above-mentioned API request log has been written.

custom log structure

Of course, during use, we can also set the log data structure sent to Kafka through the metadata configuration provided by the kafka-logger plugin. By setting log_format data, you can control the type of data sent.

For example, $host , $time_iso8601 , etc. in the following data are all from the built-in variables provided by Nginx; it also supports the variable configuration provided by APISIX such as $route_id and $service_id .

curl -XPUT 'http://127.0.0.1:9080/apisix/admin/plugin_metadata/kafka-logger' \
--header 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
--header 'Content-Type: application/json' \
--data-raw '{
    "log_format": {
        "host": "$host",
        "@timestamp": "$time_iso8601",
        "client_ip": "$remote_addr",
        "route_id": "$route_id"
    }
}'

By sending a request for a simple test, you can see that the above log structure settings have taken effect. At present, Apache APISIX provides a variety of log format templates, which have great flexibility in configuration. For more log format details, please refer to official document .

close plugin

After use, just remove the kafka-logger plug-in related configuration in the routing configuration and save it, then the plug-in on the routing can be closed. Thanks to the dynamic advantages of Apache APISIX, the process of opening and closing the plug-in does not need to restart Apache APISIX, which is very convenient.

$ curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "methods": ["GET"],
    "uri": "/hello",
    "plugins": {},
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "127.0.0.1:1980": 1
        }
    }
}'

Summarize

This article introduces the function preview and usage steps of the kafka-logger plugin. For more information about the kafka-logger plugin description and complete configuration list, you can refer to official document .

Currently, we are also developing other log plugins to integrate with more related services. If you are interested in such an integration project, feel free to start a discussion at GitHub Discussions , or communicate via the mailing list .