Microservices have everything from code to k8s deployment (13. Service monitoring)

We use a series to explain the complete practice of microservices from requirements to online, from code to k8s deployment, from logging to monitoring, etc.

The whole project uses microservices developed by go-zero, which basically includes go-zero and some middleware developed by related go-zero authors. The technology stack used is basically the self-developed components of the go-zero project team, basically go -zero the whole family bucket.

Actual project address: https://github.com/Mikaelemmmm/go-zero-looklook

1 Overview

A good service must be monitored in time. In go-zero-looklook, we use the currently popular prometheus as a monitoring tool, and then use grafana to display

go-zero has integrated prometheus for us in the code

 // StartAgent starts a prometheus agent.
func StartAgent(c Config) {
  if len(c.Host) == 0 {
    return
  }

  once.Do(func() {
    enabled.Set(true)
    threading.GoSafe(func() {
      http.Handle(c.Path, promhttp.Handler())
      addr := fmt.Sprintf("%s:%d", c.Host, c.Port)
      logx.Infof("Starting prometheus agent at %s", addr)
      if err := http.ListenAndServe(addr, nil); err != nil {
        logx.Error(err)
      }
    })
  })
}

No matter when we start api, rpc will start an additional goroutine to provide prometheus services

[Note] If a service managed by serviceGroup like our previous order-mq is used, it can only be called explicitly in the startup file main, api and rpc are not required, and the configuration is the same

 package main
.....
func main() {
  ....
  // log、prometheus、trace、metricsUrl.
  if err := c.SetUp(); err != nil {
    panic(err)
  }

  ......
}

2. Realize

2.1 Configure prometheus and grafana

In the docker-compose-env.yml file under the project

Let's take a look at the prometheus configuration file at deploy/prometheus/server/prometheus.yml

 global:
  scrape_interval:
  external_labels:
    monitor: 'codelab-monitor'

# 这里表示抓取对象的配置
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s  #重写了全局抓取间隔时间，由15秒重写成5秒
    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: 'order-api'
    static_configs:
      - targets: ['order-api:9091']
        labels:
          job: order-api
          app: order-api
          env: dev
  - job_name: 'order-rpc'
    static_configs:
      - targets: ['order-rpc:9091']
        labels:
          job: order-rpc
          app: order-rpc
          env: dev
  - job_name: 'order-mq'
    static_configs:
      - targets: ['order-mq:9091']
        labels:
          job: order-mq
          app: order-mq
          env: dev
  - job_name: 'usercenter-api'
    static_configs:
      - targets: ['usercenter-api:9091']
        labels:
          job: usercenter-api
          app: usercenter-api
          env: dev
  - job_name: 'usercenter-rpc'
    static_configs:
      - targets: ['usercenter-rpc:9091']
        labels:
          job: usercenter-rpc
          app: usercenter-rpc
          env: dev
  - job_name: 'travel-api'
    static_configs:
      - targets: ['travel-api:9091']
        labels:
          job: travel-api
          app: travel-api
          env: dev
  - job_name: 'travel-rpc'
    static_configs:
      - targets: ['travel-rpc:9091']
        labels:
          job: travel-rpc
          app: travel-rpc
          env: dev
  - job_name: 'payment-api'
    static_configs:
      - targets: ['payment-api:9091']
        labels:
          job: payment-api
          app: payment-api
          env: dev
  - job_name: 'payment-rpc'
    static_configs:
      - targets: ['payment-rpc:9091']
        labels:
          job: payment-rpc
          app: payment-rpc
          env: dev
  - job_name: 'mqueue-rpc'
    static_configs:
      - targets: ['mqueue-rpc:9091']
        labels:
          job: mqueue-rpc
          app: mqueue-rpc
          env: dev
  - job_name: 'message-mq'
    static_configs:
      - targets: ['message-mq:9091']
        labels:
          job: message-mq
          app: message-mq
          env: dev
  - job_name: 'identity-api'
    static_configs:
      - targets: ['identity-api:9091']
        labels:
          job: identity-api
          app: identity-api
          env: dev
  - job_name: 'identity-rpc'
    static_configs:
      - targets: [ 'identity-rpc:9091' ]
        labels:
          job: identity-rpc
          app: identity-rpc
          env: dev

2.2 Business Configuration

In terms of implementation, our business does not need to add any code (except for services managed by serviceGroup)

We only need to configure it in the business configuration file, let's take usercenter as an example

1) api

2) rpc

3) mq (serviceGroup)

[Note] (emphasis again) If a service like our previous order-mq is managed by serviceGroup, it needs to be displayed and called in the startup file main, api and rpc are not required

 package main
.....
func main() {
  ....
  // log、prometheus、trace、metricsUrl.
  if err := c.SetUp(); err != nil {
    panic(err)
  }

  ......
}

2.3 View

Visit http://127.0.0.1:9090/ , click "Status" on the menu above, and then click Targets, the blue one means the startup is successful, and the red one is the unsuccessful startup

2.4 Configure grafana

Visit http://127.0.0.1:3001 , the default account and password are admin

The configuration data source is prometheus

then configure

[Note] This is configured in docker, so the url of http cannot write 127.0.0.1

Check if the configuration is successful

Configure the dashboard

then click on the first

We add a cpu indicator and enter the cpu selection below

Then we can see the monitoring metrics we want to see

3. The end

Only one indicator is demonstrated here. You can configure other indicators you want to see by yourself. At the same time, you can also add alert alarm configuration in grafana. This is not a demonstration.

project address

https://github.com/zeromicro/go-zero

Welcome go-zero and star support us!

WeChat exchange group

Follow the official account of " Microservice Practice " and click on the exchange group to get the QR code of the community group.