Easy to use Fluentd, combined with Rainbond plugin market, log collection is faster

There was an article in the past that introduced plugin log collection for EFK (Kibana + ElasticSearch + Filebeat) . The Filebeat plugin is used to forward and centralize log data and forward them to Elasticsearch or Logstash for indexing, but Filebeat, as a member of Elastic, can only be used across the entire Elastic stack.

Fluentd

Fluentd is an open source, distributed log collection system that can collect logs from different services and data sources, filter and process the logs, and distribute them to various storage and processing systems. It supports various plug-ins and data caching mechanisms, and requires very few resources. It has built-in reliability, and combined with other services, it can form an efficient and intuitive log collection platform.

This article introduces the use of the Fluentd plugin in Rainbond to collect business logs and output to multiple different services.

1. Integrated Architecture

When collecting component logs, you only need to enable the Fluentd plugin in the component. This article will demonstrate the following two methods:

1. Kibana + ElasticSearch + Fluentd
2. Minio + Fluentd

We made Fluentd into Rainbond 一般类型插件 . After the application starts, the plug-in also starts and automatically collects logs and outputs them to multiple service sources. The whole process is non-intrusive to the application container and has strong scalability.

Second, the plug-in principle analysis

New in Rainbond V5.7.0: Install plugins from open source app stores . The plugins in this article have been released to open source app stores. When we use them, we can install them with one click, and modify the configuration files as needed.

The Rainbond plug-in system is a part of the Rainbond application model. The plug-ins are mainly used to implement the extended operation and maintenance capabilities of the application container. Because the implementation of operation and maintenance tools has a large commonality, the plug-in itself can be reused. Plug-ins must be bound to the application container to have runtime status to implement an operation and maintenance capability, such as performance analysis plug-ins, network governance plug-ins, and initialization type plug-ins.

In the process of making Fluentd plug-ins, general type plug-ins are used, which can be understood as one POD to start two Containers. Kubernetes natively supports starting multiple Containers in one POD, but the configuration is relatively complicated. In Rainbond, the plug-ins are implemented to enable users to operate Simpler.

3. EFK log collection practice

The Fluentd-ElasticSearch7 output plugin writes log records to Elasticsearch. By default, it creates records using the bulk API, which performs multiple indexing operations in a single API call. This reduces overhead and can greatly improve indexing speed.

3.1 Operation steps

Both applications (Kibana + ElasticSearch) and plugins (Fluentd) can be deployed with one click through open source app stores.

1. Docking with open source app stores
2. Search the app store for elasticsearch and install the 7.15.2 version.
3. Team View -> Plugins -> Install Fluentd-ElasticSearch7 plugins from app store

4. Create a component based on an image, the image uses nginx:latest , and the mount storage var/log/nginx . Here we use Nginx:latest as a demonstration

   • After the storage is mounted in the component, the plugin will also mount the storage and access the log files generated by Nginx.
5. Open the plug-in in the Nginx component, you can modify the Fluentd configuration file according to your needs, please refer to the introduction section of the configuration file below.

6. Add ElasticSearch dependencies and connect Nginx to ElasticSearch, as shown below:

7. Visit the Kibana panel, go to Stack Management -> Data -> Index Management, you can see that the existing index name is fluentd.es.nginx.log ,
8. Visit the Kibana panel, go to Stack Management -> Kibana -> Index Mode, and create an index mode.
9. Go to Discover, and the log is displayed normally.

3.2 Introduction to configuration files

The configuration file refers to the Fluentd documentation output_elasticsearch .

 <source>
  @type tail
  path /var/log/nginx/access.log,/var/log/nginx/error.log
  pos_file /var/log/nginx/nginx.access.log.pos
  <parse>
    @type nginx
  </parse>
  tag es.nginx.log
</source>

<match es.nginx.**>
  @type elasticsearch   
  log_level info          
  hosts 127.0.0.1
  port 9200
  user elastic
  password elastic
  index_name fluentd.${tag}
  <buffer>
    chunk_limit_size 2M
    queue_limit_length  32
    flush_interval 5s
    retry_max_times 30
  </buffer>
</match>

Configuration item explanation:

Input source for \<source>\</source> logs:

configuration item	explain
@type	Collection log type, tail indicates incremental read log content
path	Log path, multiple paths can be separated by commas
pos_file	Used to mark the path where the position file has been read
\<parse>\</parse>	For log format parsing, write the corresponding parsing rules according to your own log format.

The output of the \<match>\</match> log:

configuration item	explain
@type	Type of service output to
log_level	设置输出日志的级别为info；支持的日志级别有： fatal , error , warn , info , debug , trace .
hosts hosts	address of elasticsearch
port	port of elasticsearch
user/password	Username/password used by elasticsearch
index_name	index defined name
\<buffer>\</buffer>	The log buffer is used to cache log events and improve system performance. Memory is used by default, and file files can also be used
chunk_limit_size	Maximum size of each block: Events will be written in blocks until the size of the block becomes this size, the memory defaults to 8M, and the file is 256M
queue_limit_length	The queue length limit for this buffer plugin instance
flush_interval	Buffer log flush event, the default is to flush the output once every 60s
retry_max_times	Maximum number of times to retry failed block output

The above are just some of the configuration parameters, other configurations can be customized with the official website documentation.

Fourth, Fluentd + Minio log collection practice

The Fluentd S3 output plugin writes log records to standard S3 object storage services such as Amazon, Minio.

4.1 Operation steps

Both apps (Minio) and plugins (Fluentd S3) can be deployed with one click through the open source app store.

1. Docking with open source app stores. Search for minio in the open source app store, and install 22.06.17 version.
2. Team View -> Plugins -> Install Fluentd-S3 plugin from app store.

3. Access Minio 9090 port, the user password is obtained from Minio Components -> Dependencies.

   • Create Bucket with custom name.

   • Go to Configurations -> Region and set Service Location

     • In the configuration file of the Fluentd plugin s3_region default is en-west-test2 .

4. Create a component based on an image, use the image nginx:latest and mount the storage var/log/nginx . Here we use Nginx:latest as a demonstration

   • After the storage is mounted in the component, the plugin will also mount the storage and access the log files generated by Nginx.
5. Enter the Nginx component, enable the Fluentd S3 plugin, and modify the s3_bucket s3_region in the configuration file

6. Establish dependencies, Nginx components depend on Minio, and update the components to make them take effect.

7. Access the Nginx service and let it generate logs, which can be seen in Minio's Bucket in a few moments.

4.2 Introduction to configuration files

For configuration files, refer to the Fluentd documentation Apache to Minio .

 <source>
  @type tail
  path /var/log/nginx/access.log
  pos_file /var/log/nginx/nginx.access.log.pos
  tag minio.nginx.access
  <parse>
    @type nginx
  </parse>
</source>

<match minio.nginx.**>
  @type s3        
  aws_key_id "#{ENV['MINIO_ROOT_USER']}"
  aws_sec_key "#{ENV['MINIO_ROOT_PASSWORD']}"
  s3_endpoint http://127.0.0.1:9000/
  s3_bucket test
  s3_region en-west-test2
  time_slice_format %Y%m%d%H%M 
  force_path_style true
  path logs/
  <buffer time>
    @type file
    path /var/log/nginx/s3
    timekey 1m                 
    timekey_wait 10s            
    chunk_limit_size 256m       
  </buffer>
</match>

Configuration item explanation:

Input source for \<source>\</source> logs:

configuration item	explain
@type	Collection log type, tail indicates incremental read log content
path	Log path, multiple paths can be separated by commas
pos_file	Used to mark the path where the position file has been read
\<parse>\</parse>	For log format parsing, write the corresponding parsing rules according to your own log format.

The output of the \<match>\</match> log:

configuration item	explain
@type	Type of service output to
aws_key_id	Minio Username
aws_sec_key	Minio password
s3_endpoint	Minio access address
s3_bucket	Minio bucket name
force_path_style	Prevent AWS SDK from breaking endpoint URLs
time_slice_format	Add this timestamp to every filename
\<buffer>\</buffer>	The log buffer is used to cache log events and improve system performance. Memory is used by default, and file files can also be used
timekey	Accumulated chunks are refreshed every 60 seconds
timekey_wait	Wait 10 seconds to refresh
chunk_limit_size	Maximum size of each block

at last

The Fluentd plugin can flexibly collect business logs and output to multiple services, and combined with the one-click installation of the Rainbond plugin market, it makes our use easier and faster.

At present, the Flunetd plugins in the Rainbond open source plugin application market are only Flunetd-S3 Flunetd-ElasticSearch7 , welcome to contribute plugins!