Ecological expansion is in progress! Apache APISIX integration with Splunk HTTP Event Collector

With the continuous iteration of technology and the continuous evolution of enterprise architecture, the complexity of the system is getting higher and higher. As the "raw material" for analysis and observation, the to support and be compatible with different analysis will greatly reduce costs for users in the process of model selection and post-operation and maintenance. Log-based analysis and observation play a very important role as the cornerstone of ensuring system stability.

As a high-performance API gateway, Apache APISIX not only has a good performance in performance, but also supports most mainstream open source and commercial log solutions through communication and co-construction with community users in the operation and maintenance of data and logs. including: HTTP Logger , TCP Logger , Kafka used to live Logger , UDP Logger , RocketMQ Logger , SkyWalking Logger , Aliyun Cloud Logging (SLS) , Google Cloud Logging and so on.

Recently, through the co-construction support of the community, a new member has been added to the Logger family bucket of Apache APISIX: Splunk HEC Logging . This article will show you how to configure and use the Splunk HEC service in Apache APISIX.

About Splunk HTTP Event Collector

Splunk is a full-text search engine for machine data, which can be used to collect, index, search and analyze various application data. According to the search engine ranking of DB Engines , Splunk currently ranks second and is a widely used search engine. Full text search software. Splunk, like ElasticSearch, is a near real-time stream of data that provides uninterrupted search results.

Splunk HTTP Event Collector (HEC) is an HTTP event collector provided by Splunk, which mainly provides the ability to send data and application events to Splunk in the HTTP(S) protocol.

About the Splunk HEC Logging plugin

splunk-hec-logging plugin is used to forward Apache APISIX request logs to Splunk for analysis and storage. After enabling this plugin, Apache APISIX will obtain the request context information in the Log stage, serialize it into Splunk Event Data format and submit it to the batch queue. When the maximum processing capacity per batch of the batch queue, or the maximum time to flush the buffer, is triggered, the data in the queue is submitted to Splunk HEC.

How to use the Splunk HEC Logging plugin

Splunk configuration steps

Deploy Splunk Enterprise

Please refer to Splunk's official installation guide for deployment. This article will demonstrate the deployment through Docker.

The Docker command parameters are as follows:

docker run -p 18088:8088 -p 18000:8000 \    # 8088为HEC端口，8000为管理后台端口
  -e "SPLUNK_PASSWORD=your-password" \      # 管理后台登录密码
  -e "SPLUNK_START_ARGS=--accept-license" \ # 接受许可证条款（Splunk默认将提供一张Enterprise Trial License）
  -e "SPLUNK_HEC_TOKEN=your-hec-token" \    # 设置默认HEC令牌，配置此项后将创建一个默认的HEC
  -itd --rm --name splunk-example splunk/splunk:latest

For specific definitions of command parameters, please refer to: Docker Splunk document .

Configure Splunk HEC

The default HEC has been configured and created in Docker, and the process of creating HEC will not be repeated here. For the specific manual creation process, please refer to the document: Set up and use HTTP Event Collector in Splunk Web .

Log into Splunk Enterprise and check HEC

Access Docker's mapped port through a browser. Because it is necessary to map the 18000 port of the management background to the 8000 port of the host, you can access it in the browser through the "loopback address plus port" method on the host during operation. For example: http://127.0.0.1:18000 , the default username for login is admin, and the password is the value of SPLUNK_PASSWORD set in the environment variable of the above example.

As shown in the figure below, the login is successful.

Click "Settings > Data Inputs" at the top right of the interface to check whether the default HEC is set successfully:

In the Inputs column of HTTP Event Collector, we can already see the number of HECs, indicating that the setting is successful.

At this point, you can click HTTP Event Collector to enter the HEC details list to view the Token information of HECs.

Token Values is the value of SPLUNK_HEC_TOKEN configured in the Docker environment variable above.

Apache APISIX configuration steps

Enable plugin

Run the following command to enable splunk-hec-logging plugin.

curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "plugins":{
        "splunk-hec-logging":{
            "endpoint":{
                // HEC 端点地址
                "uri":"http://127.0.0.1:18088/services/collector",
                // HEC Token
                "token":"BD274822-96AA-4DA6-90EC-18940FB2414C"
            },
            // // 刷新批处理队列缓冲区的最大时间（以秒为单位）
            "inactive_timeout":2,
             // 每个批处理队列最大容纳日志条目数
            "batch_max_size":10
        }
    },
    "upstream":{
        "type":"roundrobin",
        "nodes":{
            "127.0.0.1:1980":1
        }
    },
    "uri":"/splunk.do"
}'

The plug-in parameter descriptions are shown in the following table.

Is the name required default value description endpoint is Splunk HEC endpoint configuration information endpoint.uri is Splunk HEC event collection APIendpoint.token is Splunk HEC identity token endpoint.channel no Splunk HEC sending channel identifier, reference: About HTTP Event Collector Indexer Acknowledgment endpoint.timeout no 10 Splunk HEC data submission timeout (in seconds) ssl_verify no TRUE to enable SSL verification, refer to: OpenResty documentation max_retry_count no 0 maximum number of retries before removing from processing pipeline retry_delay no 1 if execution fails , the number of seconds that process execution should be delayed buffer_duration no 60 the maximum age in seconds the oldest entry in the batch must be processed first inactive_timeout no 5 the maximum time in seconds to flush the buffer batch_max_size no 1000 per batch The maximum number of entries the processing queue can hold

send request

Run the following command to send a request to Splunk.

$ curl -i http://127.0.0.1:9080/splink.do
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Date: Fri, 10 Dec 2021 09:57:52 GMT
Server: APISIX/2.11.0

Hello, Splunk HEC Logging

verification log

Log in to the Splunk console and click Search & Reporting.

Enter: source="apache-apisix-splunk-hec-logging" in the search input box to query the sent request log.

Disable plugin

Just remove the splunk-hec-logging related configuration.

curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri":"/logging.do",
    "upstream":{
        "type":"roundrobin",
        "nodes":{
            "127.0.0.1:1980":1
        }
    },
    "plugins":{
    }
}'

Summarize

Currently, Apache Discussion APISIX is also developing other plugins to support the integration of more services. If you are interested in this, please feel free to initiate discussions on GitHub , or communicate via the mailing list .

Related Reading

• Apache integrates Kafka for efficient real-time log monitoring16217386a669fa
• Apache APISIX joins hands with RocketMQ for real-time API log monitoring function
• news spreads frequently! Apache APISIX now supports Google Cloud Logging
• strong alliance! APISIX integrates to create comprehensive log processing16217386a66ae4