jmxtrans+influxdb+grafana监控zookeeper实战

序

本文主要研究一下如何使用jmxtrans+influxdb+granfa监控zookeeper

配置zookeeper jmx

在conf目录下新增zookeeper-env.sh，并使用chmod +x赋予执行权限，内容如下

JMXLOCALONLY=false
JMXDISABLE=false
JMXPORT=8999
JMXAUTH=false
JMXSSL=false

• 指定JMXPORT为8999

搭建influxdb及grafana

这里使用docker构建，如下

docker network create monitoring-network
docker run -d -p 8083:8083 -p 8086:8086 \
    --net monitoring-network  \
    --name influxdb  \
    influxdb:1.5.4
curl -POST http://192.168.99.100:8086/query --data-urlencode "q=CREATE DATABASE zookeeper"
docker run -d -p 3000:3000 --net monitoring-network --name grafana grafana/grafana:5.2.4
curl 'http://admin:admin@192.168.99.100:3000/api/datasources' -X POST \
-H 'Content-Type: application/json;charset=UTF-8' \
--data-binary '{"name":"influx","type":"influxdb","url":"http://192.168.99.100:8086",
"access":"proxy","isDefault":true,"database":"zookeeper","user":"admin","password":"admin"}'

• 创建monitoring-network，让docker中的influxdb与grafana连通，然后使用rest api创建zookeeper数据库
• 创建grafana并指定net为monitoring-network，然后使用rest api配置数据源

jmxtrans配置

• 下载jmxtrans

wget http://central.maven.org/maven2/org/jmxtrans/jmxtrans/270/jmxtrans-270-all.jar

• 下载执行脚本

wget -q https://raw.githubusercontent.com/jmxtrans/jmxtrans/master/jmxtrans/jmxtrans.sh
chmod +x jmxtrans.sh

• 配置zookeeper.json

{
  "servers": [
    {
      "port": 8999,
      "host": "localhost",
      "alias": "zk",
      "queries": [
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*",
          "attr": [
            "QuorumSize"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*",
          "attr": [
            "MaxClientCnxnsPerHost",
            "MaxRequestLatency",
            "AvgRequestLatency",
            "MinRequestLatency",
            "MaxSessionTimeout",
            "MinSessionTimeout",
            "OutstandingRequests",
            "PacketsReceived",
            "PacketsSent",
            "PendingRevalidationCount",
            "TickTime"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*,name3=InMemoryDataTree",
          "attr": [
            "NodeCount",
            "WatchCount"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*,name3=Connections,name4=*,name5=*",
          "resultAlias": "Clients",
          "attr": [
            "AvgLatency",
            "LastLatency",
            "MaxLatency",
            "MinLatency",
            "OutstandingRequests",
            "PacketsReceived",
            "PacketsSent"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=StandaloneServer_port*",
          "resultAlias": "Clients",
          "attr": [
            "AvgLatency",
            "LastLatency",
            "MaxLatency",
            "MinLatency",
            "OutstandingRequests",
            "PacketsReceived",
            "PacketsSent",
            "NumAliveConnections"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "java.lang:type=Memory",
          "resultAlias": "Memory",
          "attr": [
            "HeapMemoryUsage",
            "NonHeapMemoryUsage"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "java.lang:type=OperatingSystem",
          "resultAlias": "Process",
          "attr": [
            "OpenFileDescriptorCount",
            "ProcessCpuLoad"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "java.lang:type=Threading",
          "resultAlias": "Threading",
          "attr": [
            "ThreadCount"
          ]
        }
      ],
      "numQueryThreads": 2
    }
  ]
}

放在跟jmxtrans.sh脚本同一个目录下即可自动加载

• 启动

JAR_FILE=jmxtrans-270-all.jar ./jmxtrans.sh start

• 日志输出实例

2018-10-04 11:28:30 [main] level org.quartz.impl.StdSchedulerFactory [StdSchedulerFactory.java:1179] - Using default implementation for ThreadExecutor
2018-10-04 11:28:30 [main] level org.quartz.core.SchedulerSignalerImpl [SchedulerSignalerImpl.java:60] - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:229] - Quartz Scheduler v.1.8.6 created.
2018-10-04 11:28:30 [main] level org.quartz.simpl.RAMJobStore [RAMJobStore.java:139] - RAMJobStore initialized.
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:255] - Scheduler meta-data: Quartz Scheduler (v1.8.6) 'ServerScheduler' with instanceId 'Mars.local1538623710548'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

2018-10-04 11:28:30 [main] level org.quartz.impl.StdSchedulerFactory [StdSchedulerFactory.java:1324] - Quartz scheduler 'ServerScheduler' initialized from an externally opened InputStream.
2018-10-04 11:28:30 [main] level org.quartz.impl.StdSchedulerFactory [StdSchedulerFactory.java:1328] - Quartz scheduler version: 1.8.6
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:2267] - JobFactory set to: com.googlecode.jmxtrans.guice.GuiceJobFactory@263f04ca
2018-10-04 11:28:30 [main] level com.googlecode.jmxtrans.JmxTransformer [JmxTransformer.java:177] - Starting Jmxtrans on : .
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:519] - Scheduler ServerScheduler_$_Mars.local1538623710548 started.
2018-10-04 11:28:30 [main] level com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory [InfluxDbWriterFactory.java:121] - Result Tags to write set to: [ResultAttribute(name=typeName), ResultAttribute(name=objDomain), ResultAttribute(name=className), ResultAttribute(name=attributeName)]
2018-10-04 11:28:30 [main] level com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory [InfluxDbWriterFactory.java:102] - Connecting to url: http://192.168.99.100:8086/ as: username: admin
2018-10-04 11:28:31 [main] level com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory [InfluxDbWriterFactory.java:121] - Result Tags to write set to: [ResultAttribute(name=typeName), ResultAttribute(name=objDomain), ResultAttribute(name=className), ResultAttribute(name=attributeName)]

查询influxdb数据

docker exec -it influxdb influx
Connected to http://localhost:8086 version 1.5.4
InfluxDB shell version: 1.5.4
> use zookeeper
Using database zookeeper
> show MEASUREMENTS
name: measurements
name
----
Clients
Memory
Process
Threading
> show series from "Memory"
key
---
Memory,attributeName=HeapMemoryUsage,className=sun.management.MemoryImpl,hostname=zk,objDomain=java.lang,typeName=type\=Memory
Memory,attributeName=NonHeapMemoryUsage,className=sun.management.MemoryImpl,hostname=zk,objDomain=java.lang,typeName=type\=Memory

• 之后通过select * from "Memory"可以查看具体指标数据
• 查询到有数据之后，就可以在grafana上面进行配置可视化，然后进行监控

小结

对于zookeeper、kafka之类应用来说，内置了jmx，因而其监控可以通过jmxtrans进行指标收集转换，然后输出到influxdb或者graphite或者prometheus等，最后通过grafana进行可视化及监控报警。

doc

• jmxtrans
• jmxtrans-270-all.jar
• How do i enable remote jmx with port in zookeeper zkServer.cmd
• Monitoring Apache Kafka with Grafana / InfluxDB via JMX