Rainbond integrates ELK/EFK through plug-ins to realize log collection

Preface

ELK is an acronym for three open source projects: Elasticsearch, Logstash, and Kibana. But the FileBeat that appeared later can completely replace Logstash's data collection function, and it is also relatively lightweight. This article will introduce EFK: Elasticsearch, Filebeat and Kibana

Elasticsearch: Distributed search and analysis engine, featuring high scalability, high reliability, and easy management. Based on Apache Lucene, it can perform near real-time storage, search and analysis operations on large-capacity data. It is usually used as a basic search engine for some applications, giving it a complex search function;

Kibana: Data analysis and visualization platform. Use with Elasticsearch to search, analyze and display the data in statistical charts;

Filebeat: Filebeat is a lightweight transporter for forwarding and centralizing log data. Filebeat is installed on your server as an agent, monitors the log file or location you specify, collects log events, and forwards them to Elasticsearch or Logstash for indexing.

Through this article, learn how to collect application logs and send them to Elasticsearch by opening the FileBeat plug-in for applications running on Rainbond.

Integrated architecture

When collecting logs, you need to enable the FileBeat plug-in in the application to collect. FileBeat collects logs in three ways:

1. Specify log path
2. Collect all container logs
3. Specify Label to automatically discover

This article uses the specified log path for collection. In this way, we can customize the rules for collecting logs.

We made FileBeat into Rainbond's general type plug-in . After the application is started, the plug-in is also started and automatically collects logs and sends it to Elasticsearch. The whole process is non-intrusive to the application container and has strong scalability. A similar method can be used for docking with other log collections. Users can connect with different log collection tools by replacing plug-ins.

The following figure shows the structure of Rainbond using the FileBeat plugin to collect application logs and send them to Elasticsearch.

Analysis of plug-in implementation principle

The Rainbond plug-in system is a part of the Rainbond application model. The plug-in is mainly used to implement application container extension operation and maintenance capabilities. Since the implementation of operation and maintenance tools has greater commonality, the plug-in itself can be reused. The plug-in must be bound to the application container to have a runtime state to achieve a kind of operation and maintenance capabilities, such as performance analysis plug-ins, network management plug-ins, and initialization type plug-ins.

The runtime environment of the plug-in with runtime is consistent with the bound components from the following aspects:

• a crucial feature. The network space is consistent so that plug-ins can bypass monitoring and intercept component network traffic, and set component local domain name resolution.
• storage persistent space enables file exchange between plug-ins and components through persistent directories.
• environment variable This feature allows the plug-in to read the component's environment variables.

In the process of making the FileBeat plug-in, the general type plug-in , which can be understood as a POD to start two Containers. Kubernetes natively supports multiple Containers in one POD, but the configuration is relatively complicated. In Rainbond, the plug-in is implemented. User operation is simple.

One-click installation of EK via Rainbond App Store

We have made elasticsearch + Kibana as an application and released it to the application market. Users can install it based on the open source application store with one click.

1. Install Rainbond
2. Search for elasticsearch in the open source application store, click install to install it with one click;

3. elasticsearch enables the xpack security module by default to protect our cluster, so we need an initial password. We enter the elasticsearch Web terminal and execute the commands shown below, and run the bin/elasticsearch-setup-passwords command in the Web terminal to generate the default user name and password:

bin/elasticsearch-setup-passwords 参数
auto 自动生成
interactive 手动填写

4. Entering Kibana assembly environment variable, modify the default connection elasticsearch environment variables ELASTICSEARCH_PASSWORD .

Collect application logs

Use Nginx as the demo application of this article, and use mirroring to create components on Rainbond,

• Mirror address: nginx:latest
• Mount storage: /var/log/nginx , make the Nginx log persistent, and the Filebeat plug-in can read the log file.

Make FileBeat plug-in

Click on the plugin on the Rainbond team interface to enter the plugin interface, click New Plugin to create a general type of plugin.

• Mirror address: docker.elastic.co/beats/filebeat:7.15.2
• Other customization is fine.

Create a plug-in and build it. After the build is successful, we will enable the FileBeat plug-in in the plug-in of the Nginx component.

Nginx configuration environment components, add the following FileBeat profile, more configuration refer official document

• Configuration file mounting path: /usr/share/filebeat/filebeat.yml
• Profile permissions: 644

filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/*.log
output.elasticsearch:
  hosts: '127.0.0.1:9200'
  username: "elastic"
  password: "elastic"

Establish dependency relationship

Establish a dependency relationship between Nginx and 127.0.0.1 , so that it can communicate with elasticsearch through the address 061cc1c98054d6, and update the Nginx component to make the dependency take effect.

Visit Kibana

Kibana is finished by default

1. Click Stack Management > Index Management, you can see that our filebeat index already exists.
2. Stack Management > Index mode, create filebeat index mode.
3. Discover can see the log information on the 061cc1c9805781 page.

Summarize

The Rainbond-based plug-in mechanism combined with EFK allows users to quickly collect application logs for analysis EFK , and flexibly replace the FileBeat Logstash .

In addition, Rainbond's plug-in mechanism is open, and application management functions are extended through the plug-in mechanism, such as network management and data backup plug-ins. In the case of no intrusion into the original application logic, network management plug-ins can be used Analyze the performance of the service and connect to log collection systems such as ELK; for components such as databases, use the backup plug-in to back up the data.

About Rainbond

Rainbond is an open source cloud native application management platform. It is simple to use, does not need to understand containers and Kubernetes, supports the management of multiple Kubernetes clusters, and provides full lifecycle management of enterprise-level applications. Its functions include application development environment, application market, and microservice architecture. , Application continuous delivery, application operation and maintenance, application-level multi-cloud management, etc.

Github：https://github.com/goodrain/rainbond

Official website: https://www.rainbond.com

WeChat group: please search and add group assistant WeChat wylhzmyj

31096419 group: please search for Dingding group number 061cc1c9805a52