APO 集成生态exporter一键完成指标采集

头像头饰
云观秋毫
     浙江
    阅读 9 分钟
    头图
    Metrics 作为可观测性领域的三大支柱之一，Metrics数据采集显得尤为重要。传统的prometheus工具采集指标，需要指定路径抓取，当指标越来越多配置会显得复杂。同时prometheus只能采集指定的指标，当用户需要节点系统相关、中间件等指标还需要引进额外组件。久而久之采集指标配置难以维护。

    APO 利用Grafana-Alloy完成指标采集

    APO 为了用户更好地一键采集各类指标，选择 Grafana-Alloy 作为APO的指标采集器，兼容OpenTelemtry生态，集成到 APO OneAgent之中，APO OneAgent负责采集所有指标，发送至APO-Server，存储至Victoria-Metrics, APO-front负责展示所有指标。当需要额外采集数据，只需配置OneAgent中Alloy数据采集源，无需更改其他组件，配置灵活，简单易懂。

    APO 指标采集配置步骤

    安装APO-Agent之时，已经安装自带安装了grafana-Alloy。APO启动之后 APO Server并对外提供服务，OneAgent抓取指标，然后发送到 Server，可以在APO Front中的Grafana查看数据。

    当用户想要修改指标采集配置，修改 apo-grafana-alloy-config ConfigMap即可(虚机环境下修改apo配置文件config/grafana-alloy/config.alloy)

    采集的配置步骤如下：

    1. 配置APO-server地址
    2. 配置apo-grafana-alloy-config文件
    3. grafana查询指标

    APO server地址配置

    首先需要配置APO Server地址，OneAgent采集指标后将数据发送到APO Server

    otelcol.receiver.prometheus "default" {
  output {
   metrics = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "<host-ip>:<port>"
    tls {
        insecure = true
        insecure_skip_verify = true
    }
  }
}

    配置说明：其中 receiver 接收 prometheus 指标，转换成 otel 格式，然后exporter导出发送至APO-Server

    APO缺采集配置

    以kubernetes环境为例，通常一个集群可能存在如下指标需要采集

    • node metrics 节点机器系统相关指标 （磁盘，cpu等信息）
    • kubelet metrics 提供 node 和 Pod 的基本运行状态和资源使用情况
    • cadvisor metrics container相关的详细资源使用和性能指标数据

    机器相关指标采集

    prometheus.exporter.unix "local_system" {
}

prometheus.scrape "scrape_metrics" {
  targets = prometheus.exporter.unix.local_system.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
  scrape_interval = "10s"
}

    该组件会采集机器上的各种资源指标

    kubernetes 指标采集

    其中 discovery.kubernetes 组件负责获取kubernetes信息, APO 这里选择获取node相关的信息

    之后采集 kubelet和 cadvisor相关的指标，由于是k8s集群，还需要配置 scheme, bearer_token_file等权限相关信息

    discovery.kubernetes "nodes" {
  role = "node"
}

prometheus.scrape "kubelet" {
  targets  = discovery.kubernetes.nodes.targets
  scheme   = "https"
  scrape_interval = "60s"
  bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  tls_config {
    insecure_skip_verify = true
  }
  clustering {
    enabled = true
  }
  forward_to = [otelcol.receiver.prometheus.default.receiver]
  job_name = "integrations/kubernetes/kubelet"
}

prometheus.scrape "cadvisor" {
  targets    = discovery.kubernetes.nodes.targets
  scheme     = "https"
  scrape_interval = "60s"
  bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  tls_config {
    insecure_skip_verify = true
  }
  clustering {
    enabled = true
  }
  forward_to = [otelcol.receiver.prometheus.default.receiver]
  job_name = "integrations/kubernetes/cadvisor"
  metrics_path = "/metrics/cadvisor"
}

    scrape指标采集

    通常用户还会部署一些自定义的探针程序，用于自定义一些监控指标

    只需指定 targets 下的 addres 用于指定采集URL, __metrics__path__自定义采集路径，默认为/metircs

    prometheus.scrape "agent_metrics" {
  targets = [
    {
      __address__ = "<scrape-path-1>:<port>",
    },
    {
      __address__ = "<scrape-path-2>:<port>",
      __metrics__path__ = "/metrics/agent"
    },
    {
      __address__ = "<scrape-path-3>:<port>",
    },
    ]
  forward_to = [otelcol.receiver.prometheus.default.receiver]
  scrape_interval = "10s"
}

    如采集APO node-agent 指标

    APO node-agent 用于采集上下游网络指标和进程启动时间指标，路径为 localhost:9500/metrics

    prometheus.scrape "agent_metrics" {
  targets = [
    {
      __address__ = "localhost:9408",
    }
    ]
  forward_to = [otelcol.receiver.prometheus.default.receiver]
  scrape_interval = "10s"
}

    一键采集中间件指标

    除了采集基本指标外，用户使用APO还可以根据自己的需求额外配置其他指标采集。

    如采集各类 中间件指标（kafka, redis, mysql, elasticsearch等）

    监控 MySQL

    1.OneAgent 的 alloy 配置文件添加如下内容，然后重启 OneAgent

    # 采集 mysql指标
prometheus.exporter.mysql "example" {
  data_source_name  = "username:password@(<mysql-url>:3306)/"
  enable_collectors = ["heartbeat", "mysql.user"]
}

prometheus.scrape "mysql" {
  targets    = prometheus.exporter.mysql.example.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
}

    2.APO Front 的 Grafana 中导入 MySQL 模版

    3.验证是否有MySQL指标数据

    监控 ElasticSearch

    1.OneAgent 的 alloy 配置文件添加如下内容，然后重启 OneAgent

    # 采集 elasticsearch指标
prometheus.exporter.elasticsearch "example" {
  address = "http://<elasticsearch-url>:9200"
  basic_auth {
    username = USERNAME
    password = PASSWORD
  }
}

prometheus.scrape "mysql" {
  targets    = prometheus.exporter.elasticsearch.example.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
}

    2.APO Front 的 Grafana 中导入 ElasticSearch 模版

    3.验证是否有ElasticSearch指标数据

    监控 Redis

    1.OneAgent 的 alloy 配置文件添加如下内容，重启OneAgent

    # 采集 redis 指标
prometheus.exporter.redis "example" {
  address = "<redis-url>:6379"
}

prometheus.scrape "redis" {
  targets    = prometheus.exporter.redis.example.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
}

    2.APO Front 的 Grafana 导入 Redis 模版

    3.验证是否有 Redis 指标数据

    监控 Kafka

    1.OneAgent 的 alloy 配置文件添加如下内容，重启OneAgent

    # 采集 kafka 指标
prometheus.exporter.kafka "example" {
  address = "<kafka-url>:9092"
}

prometheus.scrape "kafka" {
  targets    = prometheus.exporter.kafka.example.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
}

    2.APO Front 的 Grafana 导入 Kafka 模版

    3.验证是否有Kafka 指标数据

    更多指标的采集可以参考Grafana-Alloy的官方文档或者咨询我们

    参考资料

    otel-collector

    otlp-configgrpc

    victora-metrics

    Sending data via OpenTelemetry

    alloy

    discovery.kubernetes

    otel.receiver.prometheus

    prometheus

    样例配置文件

    logging {
  level  = "info"
  format = "logfmt"
}


otelcol.receiver.prometheus "default" {
  output {
   metrics = [otelcol.processor.transform.default.input]
  }
}

otelcol.processor.transform "default" {
  error_mode = "ignore"
  trace_statements {
    context = "resource"
    statements = [
     `replace_all_patterns(attributes, "key", "service\\.instance\\.id", "service_instance_id")`,
     `replace_all_patterns(attributes, "key", "service\\.name", "service_name")`,
     `replace_all_patterns(attributes, "key", "net\\.host\\.name", "net_host_name")`,
    ]
  }
  output {
    metrics = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "<host-ip>:<port>"
    tls {
        insecure = true
        insecure_skip_verify = true
    }
  }
}

prometheus.exporter.unix "local_system" {
}

prometheus.scrape "scrape_metrics" {
  targets = prometheus.exporter.unix.local_system.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
  scrape_interval = "10s"
}

prometheus.scrape "agent_metrics" {
  targets = [
    {
      __address__ = "<scrape-path-1>",
    },
    {
      __address__ = "<scrape-path-2>",
    },
    {
      __address__ = "<scrape-path-3>",
    },
    ]
  forward_to = [otelcol.receiver.prometheus.default.receiver]
  scrape_interval = "10s"
}

discovery.kubernetes "nodes" {
  role = "node"
}

prometheus.scrape "kubelet" {
  targets  = discovery.kubernetes.nodes.targets
  scheme   = "https"
  scrape_interval = "60s"
  bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  tls_config {
    insecure_skip_verify = true
  }
  clustering {
    enabled = true
  }
  forward_to = [otelcol.receiver.prometheus.default.receiver]
  job_name = "integrations/kubernetes/kubelet"
}

prometheus.scrape "cadvisor" {
  targets    = discovery.kubernetes.nodes.targets
  scheme     = "https"
  scrape_interval = "60s"
  bearer_token_file = "/var/run/secrets/kubernetes.io/serviceaccount/token"
  tls_config {
    insecure_skip_verify = true
  }
  clustering {
    enabled = true
  }
  forward_to = [otelcol.receiver.prometheus.default.receiver]
  job_name = "integrations/kubernetes/cadvisor"
  metrics_path = "/metrics/cadvisor"
}


# 采集 mysql指标
prometheus.exporter.mysql "example" {
  data_source_name  = "username:password@(<mysql-url>:3306)/"
  enable_collectors = ["heartbeat", "mysql.user"]
}

// Configure a prometheus.scrape component to send metrics to.
prometheus.scrape "mysql_metrics" {
  targets    = prometheus.exporter.mysql.example.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
}

# 采集 elasticsearch指标
prometheus.exporter.elasticsearch "example" {
  address = "http://<elasticsearch-url>:9200"
  basic_auth {
    username = USERNAME
    password = PASSWORD
  }
}

prometheus.scrape "demo" {
  targets    = prometheus.exporter.elasticsearch.example.targets
  forward_to = [otelcol.receiver.prometheus.default.receiver]
}
    服务器运维
    云观秋毫
    20 声望0 粉丝

    Kindling - OriginX 故障根因推理引擎，基于 eBPF 的自动化 Tracing 分析

    « 上一篇
    APO如何快速判断云环境网络质量是否有问题
    下一篇 »
    APO 新发版支持Skywalking Agent接入

    引用和评论

    被 1 篇内容引用
    推荐阅读
    头像
    APO v0.9.0 更新：告警分析功能免费公测；全量日志性能优化；多项问题修复

    云观秋毫阅读 39

    头像
    MySQL 回收表碎片实践教程

    MySQL技术赞 2阅读 332

    头像
    CentOS7 Docker安装Gitlab

    YYGP阅读 6.7k

    头像
    docker 使用--storage-opt参数约束容器文件系统大小

    李朝阳赞 2阅读 4.6k

    头像
    Docker 万字入门教程

    Java陈序员赞 3阅读 618

    头像
    如何免费申请公网 IP 的 SSL 证书 (无需域名)

    好心人帅崽赞 1阅读 497

    头像
    Kubernetes 网关流量管理：Ingress 与 Gateway API

    凌虚赞 1阅读 891

    0 条评论
    得票最新