轻量级日志系统Loki监控日志并告警

头像
会当凌绝顶
     甘肃
    阅读 7 分钟

    1. 简介

    通过Promtail+Loki+Grafana+Alertmanager实现对常用日志的监控，并进行告警。

    2. 系统环境

    ① Promtail：                    192.168.83.137 39090、192.168.83.137 39090
② Loki：                         192.168.83.137 39090
③ Grafana：                     192.168.83.137 33000
④ Alertmanager：                192.168.83.137 39093

    3. 实现要求

    ① 通过Promtail实现对linux系统日志、nginx日志的监控，并通过Grafana进行图形化展示
② 对监控异常日志数据配置告警。

    4. 下载

    ① Promtail

    https://github.com/grafana/loki/releases/download/v3.5.0/promtail-linux-amd64.zip
https://github.com/grafana/loki/releases/download/v3.5.0/promtail-3.5.0.x86_64.rpm

    ② Loki

    https://github.com/grafana/loki/releases/download/v3.5.0/loki-linux-amd64.zip
https://github.com/grafana/loki/releases/download/v3.5.0/loki-3.5.0.x86_64.rpm

    ③ Grafana和Alertmanager

    参考https://segmentfault.com/a/1190000046552086，不再赘述。

    5. 安装配置

    ① Promtail

    安装：

    mkdir /usr/local/share/applications/promtail-linux-amd64
unzip promtail-linux-amd64.zip -d /usr/local/share/applications/promtail-linux-amd64
cd /usr/local/share/applications/promtail-linux-amd64
vim config.yaml
    server:
  http_listen_port: 39080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://192.168.83.137:33100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - 192.168.83.138
        labels:
          job: varlogs
          host: 192.168.83.138
          __path__: /var/log/*log
          stream: stdout

  - job_name: messages
    static_configs:
      - targets:
          - 192.168.83.138
        labels:
          job: messages
          host: 192.168.83.138
          __path__: /var/log/messages
          stream: stdout

  - job_name: nginxlogs
    static_configs:
      - targets:
          - localhost
        labels:
          job: nginxlogs
          host: 192.168.83.138
          __path__: /var/log/nginx/access.log
    pipeline_stages:
      - regex:
          expression: '^(?P<remote_addr>[^ ]*) - (?P<remote_user>[^ ]*) $$time_local:[^$$]*$$ "(?P<request>[^"]*)" (?P<status>[^ ]*) (?P<body_bytes_sent>[^ ]*) "(?P<http_referer>[^"]*)" "(?P<http_user_agent>[^"]*)" "(?P<http_x_forwarded_for>[^"]*)" rt=(?P<request_time>[^ ]*) uct="(?P<upstream_connect_time>[^"]*)" uht="(?P<upstream_header_time>[^"]*)" urt="(?P<upstream_response_time>[^"]*)"$'
      - labels:
          remote_addr:
          status:
          request_time:
          upstream_connect_time:
          upstream_header_time:
          upstream_response_time:

  - job_name: secure
    static_configs:
      - targets:
          - 192.168.83.138
        labels:
          job: secure
          host: 192.168.83.138
          __path__: /var/log/secure
          stream: stdout

    启动：

    ./promtail-linux-amd64 -config.file=config.yaml

    验证：

    http://192.168.83.137:39080/targets

    创建服务：

    vim /usr/lib/systemd/system/promtail.service
    [Unit]
Description=Promtail is a logs collector built specifically for Loki.
Documentation=https://github.com/grafana/loki/
After=network.target
 
[Service]
Type=simple
WorkingDirectory=/usr/local/share/applications/promtail-linux-amd64
ExecStart=/usr/local/share/applications/promtail-linux-amd64/promtail-linux-amd64 -config.file=/usr/local/share/applications/promtail-linux-amd64/config.yaml
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

    验证服务：

    systemctl daemon-reload
systemctl start promtail
systemctl enable promtail
systemctl status promtail
netstat -lantup | grep 39080

    ② Loki

    安装：

    mkdir /usr/local/share/applications/loki-linux-amd64
unzip loki-linux-amd64.zip -d /usr/local/share/applications/loki-linux-amd64
cd /usr/local/share/applications/loki-linux-amd64
vim config.yaml
    # Loki全局配置

# Loki全局配置

# 认证开关，默认关闭（生产环境中建议开启）
auth_enabled: false

server:
  # HTTP服务器监听地址
  http_listen_address: 0.0.0.0
  # HTTP服务器监听端口
  http_listen_port: 33100
  grpc_listen_port: 39096
  # 日志级别，生产环境中建议使用'info'
  log_level: info
  # gRPC服务最大并发流数
  #grpc_server_max_concurrent_streams: 1000

common:
  # 实例地址
  instance_addr: 127.0.0.1
  # 数据存储路径前缀，建议使用持久化存储位置
  path_prefix: /data/loki
  storage:
    filesystem:
      # 分块数据目录
      chunks_directory: /data/loki/chunks
      # 规则数据目录
      rules_directory: /usr/local/share/applications/loki-linux-amd64/rules
  # 数据复制因子，生产环境中应大于1以提供冗余
  replication_factor: 1
  ring:
    kvstore:
      # 键值存储后端，开发环境中使用'inmemory'
      store: inmemory

schema_config:
  configs:
    - from: "2020-10-24"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

pattern_ingester:
  enabled: true
  metric_aggregation:
    # 指向Loki服务地址
    loki_address: 127.0.0.1:33100

ruler:
  # rules规则存储
  # 主要支持本地存储（local）和对象文件系统（azure, gcs, s3, swift）
  storage:
    type: local
    local:
      directory: /usr/local/share/applications/loki-linux-amd64/rules  # Loki告警规则存储路径
  # rules临时规则文件存储路径
  rule_path: /usr/local/share/applications/loki-linux-amd64/rules-tmp
  # rules规则加载时间
  flush_period: 1m
  # Alertmanager URL
  alertmanager_url: http://192.168.83.137:39093
  ring:
    kvstore:
      store: inmemory
  enable_api: true
  enable_alertmanager_v2: true

    启动：

    ./loki-linux-amd64 -config.file=config.yaml

    验证：

    http://192.168.83.137:33100

    创建服务：

    vim /usr/lib/systemd/system/loki.service
    [Unit]
Description=Store the logs in Loki.
Documentation=https://github.com/grafana/loki/
After=network.target
 
[Service]
Type=simple
WorkingDirectory=/usr/local/share/applications/loki-linux-amd64
ExecStart=/usr/local/share/applications/loki-linux-amd64/loki-linux-amd64 -config.file=/usr/local/share/applications/loki-linux-amd64/config.yaml
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

    验证服务：

    systemctl daemon-reload
systemctl start loki
systemctl enable loki
systemctl status loki
netstat -lantup | grep 33100

    ③ Grafana和Alertmanager

    Grafana和Alertmanager参考https://segmentfault.com/a/1190000046552086，不再赘述。

    6. 监控

    grafana 添加Prometheus源：

    Home > Connections > Add new connection,选择loki，点击 Add new data source. 点击填入loki URL:http://192.168.83.137:33100/，点击 test & save.

    添加成功后，在Home > Connections > Data sources，可看到添加的源

    7. 告警

    SSH 登录告警规则：不在指定 IP 列表内的登录尝试
    vim /usr/local/share/applications/loki-linux-amd64/rules/ssh_alerts.rule

    groups:
- name: ssh-login-alerts
  rules:
  - alert: UnauthorizedSSHLoginAttempt
    expr: |
      sum by (host, user) (
        count_over_time(
          {filename="/var/log/secure"}
          |~ "Accepted.*for.*from"
          | regexp "from (?P<ip>\\b(?:\\d{1,3}\\.){3}\\d{1,3}\\b)"
          | ip !~ "192.168.83.137|192.168.83.2"
          [5m]
        )
      ) > 0
    for: 1m
    labels:
      severity: warning
      category: security
    annotations:
      summary: "Unauthorized SSH login attempt detected (Host: {{ $labels.host }}, User: {{ $labels.user }})"
      description: "SSH login from suspicious IP (count: {{ $value }})"
      playbook: "Check /var/log/secure for details and consider blocking the source IP if malicious"

    验证

    通过非列表ip登录，触发告警

    loki
    会当凌绝顶
    9 声望3 粉丝
    « 上一篇
    Prometheus+Grafana+Alertmanager监控

    引用和评论

    推荐阅读
    头像
    高效日志管理与可视化：Loki与Grafana结合优化高频日志处理

    Swift社区阅读 735

    头像
    高并发微服务日志管理：ELK、Loki、Fluentd 终极对决与实战指南

    Swift社区阅读 649

    0 条评论
    得票最新