Synchronize server logs to SeaTable, so that logs can be better visualized and processed collaboratively

SeaTable Developer Edition is a free, easy-to-use, scalable data center product. It combines the ease of use of collaborative tables with the powerful data processing capabilities of the database. SeaTable provides Restful API, so you can easily write data through HTTP protocol anytime and anywhere.

In this article, we mainly introduce how to use SeaTable to record the error logs of various services on the server, realize the visualization of the logs, and prevent the error messages from being ignored. The overall architecture is as follows:

The example is as follows:

Features include

• The log is placed in a code block in Markdown format and recorded in the long text column of SeaTable
• We can use SeaTable's multi-view function to create multiple different views, each view includes specific filter conditions, which is convenient for us to quickly delete and view data

Let's introduce the construction process in detail below.

Use filebeat to send logs to Redis

Install filebeat

Install on the host, take Ubuntu 20.04 as an example

apt update && apt install filebeat -y

filebeat configuration file

vim /etc/filebeat/filebeat.yml

The content is as follows, modified according to the actual configuration

https://github.com/seatable/seatable-syncer/blob/main/log-sync/filebeat.yml

filebeat.inputs:

  - type: log

    paths:

      - /opt/seatable/shared/seatable/logs/dtable_web.log        # 日志路径

    tags: ["dtable-web"]                                         # 对应表格中的 Service

    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}|^[[0-9]{4}-[0-9]{2}-[0-9]{2}'

    multiline.negate: true        # 多行匹配，以日期开头的信息，例如：'2021-10-12' 或 '[2021-10-12'

    multiline.match: after

  
  - type: log

    paths:

      - /opt/seatable/shared/seatable/logs/dtable-server.log

    tags: ["dtable-server"]

    multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}|^[[0-9]{4}-[0-9]{2}-[0-9]{2}'

    multiline.negate: true

    multiline.match: after


output.redis:

  hosts: ["redis host"]

  password: "redis password"

  db: 0

  timeout: 5

  keys:

    - key: "seatable-error-logs"  # Redis key，不可修改

      when.contains:

        message: "[ERROR]"        # 匹配日志中的关键词

The above example configuration is suitable for the time format in the log to 2021-10-12 or [2021-10-12 , which can identify multi-line error logs.

Start filebeat

filebeat will send matching logs to Redis

service filebeat start

Use scripts to read messages from Redis to SeaTable

SeaTable table

1. Generate API Token
2. Create the following columns, case sensitive

name	First column	type
Service	√	Text or single selection
Time	x	Date, accurate to the minute
Log	x	Long text

script

Install on the host, the path /opt/ as an example

cd /opt/

git clone [e-mail]:seatable/seatable-syncer.git

Install Python dependencies

cd seatable-syncer/log-sync/

pip3 install -r requirements.txt

Configuration file

vim log_syncer_settings.py

The content is as follows, modified according to the actual configuration

# SeaTable

server_url = 'SeaTable 服务器地址'

api_token = '表格的 api_token'

table_name = 'table name'


# Redis

redis_host = 'redis host'

redis_db = 0

redis_port = 6379

redis_password = None

Startup script

The script reads matching logs from Redis and sends them to SeaTable

python3 log_syncer.py

log_syncer.py can handle logs starting with the following format

2020-01-01 01:02:02

2020-01-01T01:02:02

[2020-01-01 01:02:02

[2020-01-01T01:02:02

If your log format does not meet the above requirements, you can modify the file log_syncer.py by yourself.

Sharing and subsequent processing in SeaTable

After synchronizing the log to a table in SeaTable, we can easily share the log information to groups and other individuals to facilitate collaborative analysis of problems. You can also use reminder rules or automation rules to immediately notify the collaborators of log information. You can also add automatic processing rules, such as logs that meet certain conditions, plus ignorable tags, and logs that meet certain conditions, plus tags to be processed.

Summarize

With filebeat, Redis and Python scripts, we can collect logs from multiple servers into a table in SeaTable. Through the filtering conditions of the table and the multi-view function, we can easily analyze and view the log. Through the shared collaboration function, we can easily achieve multi-person collaborative processing. This article is an inspiration, I hope you will find more usages of SeaTable and improve your work efficiency!