Take you to easily get the Go microservice series in ten days (8. Service monitoring)

Preamble

We will show you a go-zero microservice example in detail through a series of articles. The whole series is divided into ten articles. The directory structure is as follows:

1. Environment construction
2. service split
3. User service
4. product service
5. Order service
6. payment service
7. RPC Service Auth Authentication
8. Service Monitoring (this article)
9. link tracking
10. Distributed transaction

I hope that through this series, you can quickly develop a mall system using go-zero in the Docker environment on the local machine, so that you can quickly get started with microservices.

Complete example code: https://github.com/nivin-studio/go-zero-mall

First, let's take a look at the overall service split diagram:

8.1 Prometheus Introduction

Prometheus is an open source monitoring and alarm system based on time series database. The basic principle is to periodically capture the status of monitored services through the HTTP protocol. Any service only needs to provide the corresponding HTTP interface can access monitoring. No need for any SDK or other integration process, the HTTP interface that outputs the monitored service information is called exporter . At present the Internet company most commonly used services are exporter can be used directly, for example Varnish , Haproxy , Nginx , MySQL , Linux System information (including disk, memory, CPU , network, etc.). Promethus has the following characteristics:

• Support for multidimensional data models (time series data consisting of metric name and key-value pairs)
• Support PromQL query language, can complete very complex query and analysis, very meaningful for chart display and alarm
• Does not rely on distributed storage, single-point servers can also be used
• Support HTTP Protocol active pull method to collect time series data
• Support PushGateway pushing time series data
• Support service discovery and static configuration to obtain monitoring targets
• Support access Grafana

8.2 go-zero use Prometheus monitoring service

go-zero framework based on an integrated Prometheus of service indicators monitoring, go-zero currently in http middleware and rpc Added monitoring of request metrics to the interceptor.

Mainly from 请求耗时 and 请求错误 two dimensions, the time-consuming request adopts Histogram the indicator type defines multiple Buckets statistics, using a request error Counter type and http metric added in path label, rpc metric added method Labels for segmented monitoring.

Next, we add Prometheus monitoring to the services implemented in the previous chapters. First, let's review the service split in Chapter 2. In order to simulate the distributed deployment of services, we start all the services in a container. service and assigned a different port number. Next, we will assign a port number for these services Prometheus to collect indicator data.

Serve	api Service port number	rpc Service port number	api Indicator collection port number	rpc Indicator collection port number
user	8000	9000	9080	9090
product	8001	9001	9081	9091
order	8002	9002	9082	9092
pay	8003	9003	9083	9093

8.2.1 Add user api service Prometheus configure

 $ vim mall/service/user/api/etc/user.yaml

 Name: User
Host: 0.0.0.0
Port: 8000

...

Prometheus:
  Host: 0.0.0.0
  Port: 9080
  Path: /metrics

8.2.2 Add user rpc service Prometheus configure

 $ vim mall/service/user/rpc/etc/user.yaml

 Name: user.rpc
ListenOn: 0.0.0.0:9000

...

Prometheus:
  Host: 0.0.0.0
  Port: 9090
  Path: /metrics

8.2.3 Add product api service Prometheus configure

 $ vim mall/service/product/api/etc/product.yaml

 Name: Product
Host: 0.0.0.0
Port: 8001

...

Prometheus:
  Host: 0.0.0.0
  Port: 9081
  Path: /metrics

8.2.4 Add product rpc service Prometheus configure

 $ vim mall/service/product/rpc/etc/product.yaml

 Name: product.rpc
ListenOn: 0.0.0.0:9001

...

Prometheus:
  Host: 0.0.0.0
  Port: 9091
  Path: /metrics

8.2.5 add order api service Prometheus configure

 $ vim mall/service/order/api/etc/order.yaml

 Name: Order
Host: 0.0.0.0
Port: 8002

...

Prometheus:
  Host: 0.0.0.0
  Port: 9082
  Path: /metrics

8.2.6 add order rpc service Prometheus configure

 $ vim mall/service/order/rpc/etc/order.yaml

 Name: order.rpc
ListenOn: 0.0.0.0:9002

...

Prometheus:
  Host: 0.0.0.0
  Port: 9092
  Path: /metrics

8.2.7 Add pay api service Prometheus configure

 $ vim mall/service/pay/api/etc/pay.yaml

 Name: Pay
Host: 0.0.0.0
Port: 8003

...

Prometheus:
  Host: 0.0.0.0
  Port: 9083
  Path: /metrics

8.2.8 add pay rpc service Prometheus configure

 $ vim mall/service/pay/rpc/etc/pay.yaml

 Name: pay.rpc
ListenOn: 0.0.0.0:9003

...

Prometheus:
  Host: 0.0.0.0
  Port: 9093
  Path: /metrics

Tip: After the configuration is modified, the service needs to be restarted to take effect.

8.2.9 Modify Prometheus Configuration

In Chapter 1 Environment Construction , we integrated the Prometheus service, and there is a prometheus.yml configuration file in the prometheus directory. We now need to modify this configuration file.

 # my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. 
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.     
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.  
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]

  # 我们自己的商城项目配置
  - job_name: 'mall'
    static_configs:
      # 目标的采集地址
      - targets: ['golang:9080']
        labels:
          # 自定义标签
          app: 'user-api'
          env: 'test'

      - targets: ['golang:9090']
        labels:
          app: 'user-rpc'
          env: 'test'

      - targets: ['golang:9081']
        labels:
          app: 'product-api'
          env: 'test'

      - targets: ['golang:9091']
        labels:
          app: 'product-rpc'
          env: 'test'

      - targets: ['golang:9082']
        labels:
          app: 'order-api'
          env: 'test'

      - targets: ['golang:9092']
        labels:
          app: 'order-rpc'
          env: 'test'

      - targets: ['golang:9083']
        labels:
          app: 'pay-api'
          env: 'test'

      - targets: ['golang:9093']
        labels:
          app: 'pay-rpc'
          env: 'test'

Tip: After the configuration file is modified, you need to restart the Prometheus service container to take effect.

8.2.10 Access Prometheus Visual interface

• In the first chapter of environment construction , we integrated the Prometheus service, and made the mapping relationship of the host port 3000 for its port number 9090 , so when browsing Enter http://127.0.0.1:3000/ to access the Prometheus interface.

• Select Status -> Targets menu, you can see the status of the acquisition target we configured and the custom label.

• After we visit the api service interface many times, select the Graph menu, and enter {path="api接口地址"} or {method="rpc接口方法"} in the query input box, that is, the command ---1fad1054acf6bc6a369--- Monitoring metrics can be viewed.

8.3 Use Grafana Visualization Prometheus indicator data

8.3.1 Add Prometheus Data Source

• In the first chapter of environment construction , we integrated the Grafana service, and made the mapping relationship of the host port 4000 for its port number 3000 , so when browsing Enter http://127.0.0.1:4000/ to access the Grafana interface in the device. Click on the left sidebar Configuration -> Data Source -> Add data source to add a data source.

• Then select Prometheus Data Source

• Fill HTTP configuration URL Address (I am here IP地址 is Prometheus where containers IP地址 ), then Click Save & test , and a prompt will appear above Data source is working , indicating that our data source has been added successfully and is working properly.

8.3.2 Added Variables for service filtering

• Click on the left sidebar Dashboard select the top right corner Dashboard settings button in Settings page select Variables -> Add variable add Variables to facilitate filtering for different tags.

• Add api_app API service name, rpc_app RPC service name variables, to filter different services. Variable data source selection Prometheus data source, use regular expressions to extract the corresponding app tags.

8.3.3 Add api interface qps dashboard

• Go back to Dashboard page and select Add panel button in the upper right corner of the page, and then select Add an empty panel to add an empty panel.

• Edit Page panel, modify the panel titled API接口QPS , in Metrics input sum(rate(http_server_requests_duration_ms_count{app="$api_app"}[5m])) by (path) to path dimension statistics api Interface qps

8.3.4 Add rpc interface qps dashboard

• Then create a new panel, modify the panel titled RPC接口QPS , in Metrics Enter sum(rate(rpc_server_requests_duration_ms_count{app="$rpc_app"}[5m])) by (method) to method statistical dimension rpc -Interface rpc qps

8.3.5 Added api interface status code dashboard

• Then create a new panel, modify the panel titled API接口状态码 , in Metrics input sum(rate(http_server_requests_code_total{app="$api_app"}[5m])) by (code) to code dimension statistics api the status code of the interface

8.3.6 Add rpc interface status code dashboard

• Then create a new panel, modify the panel titled RPC接口状态码 , in Metrics input sum(rate(rpc_server_requests_code_total{app="$rpc_app"}[5m])) by (code) to code dimension statistics rpc the status code of the interface

8.3.7 Save Dashboard

• Adjust the position of the lower panel, select the upper right corner Save dashboard button to save the dashboard.

project address

https://github.com/zeromicro/go-zero

Welcome go-zero and star support us!

WeChat exchange group

Follow the official account of " Microservice Practice " and click on the exchange group to get the QR code of the community group.