Use DevOps to deploy Java microservices in KubeSphere, configure monitoring and alerts

Author: Soy Sauce Bottle, Ctrip Backend Technical Expert, KubeSphere Community User

Develop Java microservices and introduce monitoring components

We develop Java microservices based on Spring Cloud +Nacos, and Java service development will not be described too much.

Introducing Actuator into the project

We introduce Spring Boot Actuator in the project's bom, which provides a variety of features to monitor and manage applications, either based on HTTP or JMX.

 <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

Configure Actuator

After the introduction of Actuator, in principle, we can use it without any configuration. In our project, we have made the following configurations in combination with actual needs and improved security:

 management.health.elasticsearch.enabled=false
management.endpoints.web.exposure.include=*
management.endpoints.web.base-path=/api/actuator
management.metrics.tags.application=${service.name}
management.metrics.tags.appid=${service.appid}
management.server.port=8090

• management.server.port: Enable an independent port to provide monitoring and avoid monitoring related APIs from being exposed outside the service;
• management.metrics.tags.xxx: Add custom tags to statistics;
• management.endpoints.web.exposure.include: used to include the list of endpoints we want to expose, here we set * to represent all.

Observe application monitoring data

When we run the written program, we can see data similar to the following by visiting http://localhost:8090/api/actuator/prometheus , including the tag data we added through the configuration, and the monitor we deploy will collect the data to Prometheus through the following address middle.

Application Deployment Configuration

1. Write DevOps documentation

 pipeline {
  agent {
    node {
      label 'maven'
    }
  }
    options{
      buildDiscarder(logRotator(numToKeepStr: '10'))
    }
    parameters {
        string(name:'APP_NAME',defaultValue: 'accounts-service',description:'应用名称 必须使用小写 需跟maven构建中一致')
        string(name:'PROJECT_NAMESPACE',defaultValue: 'basebiz',description:'部署项目集名称')
        string(name:'SERVICE_SRC_PATH',defaultValue: 'accounts-service-webapp',description:'war包路径')
        string(name:'PROGECT_GIT_PATH',defaultValue:'basebiz/accounts-service.git',description:'项目gitlabpath ')
        string(name:'TAG_NAME',defaultValue: '',description:'tag 发布线上必须填写 格式v20210101(v+当前日期)')
        string(name:'PODCOUNT',defaultValue: '2',description:'部署pod数量')
        string(name:'HEALTH_CHECK_URI',defaultValue: '/api/actuator/health',description:'健康检测地址')
    }
    environment {
        //构建镜像
        REGISTRY = 'hub.xxxx.cn'
        DOCKERHUB_NAMESPACE = 'app'
        DOCKER_CREDENTIAL_ID = 'dockerhub-account' //hub账号密钥
        GITHUB_CREDENTIAL_ID = 'gitlab-account' //gitlab账号密钥
        //环境部署凭证
        KUBECONFIG_CREDENTIAL_ID_DEV = 'testing-kubeconfig'
        KUBECONFIG_CREDENTIAL_ID_VIEW = 'xxxxaliyun-testing'
        KUBECONFIG_CREDENTIAL_ID_PROD = 'xxx-prod'
        DOWNLOAD_BASEDOMAIN = 'gitlab.xxxx.cn' //公共资源下载
        COMMIT_ID= sh(  returnStdout: true, script: 'git rev-parse --short HEAD').trim()

    }
 stages {
        stage ('迁出代码') {
            steps {
                checkout(scm)
            }
        }
        stage ('编译') {
            steps {
                container ('maven') {

                    //***************************************
                    //**************下载通用模版***************
                    sh 'curl -o `pwd`/start.sh https://${DOWNLOAD_BASEDOMAIN}/base/basicevn/-/raw/master/shell/springboot-start.sh'
                    sh 'curl -o `pwd`/settings.xml https://${DOWNLOAD_BASEDOMAIN}/base/basicevn/-/raw/master/setting.xml'
                    sh 'curl -o `pwd`/Dockerfile https://${DOWNLOAD_BASEDOMAIN}/base/basicevn/-/raw/master/dockerfile/javaservice/dockerfile'
                    //***************************************
                    sh 'mkdir `pwd`/yaml'
                    sh 'curl -o `pwd`/yaml/devops-java.yaml https://${DOWNLOAD_BASEDOMAIN}/base/basicevn/-/raw/master/yaml/java-service-v1.0.0.yaml'
                    sh 'mvn -Dmaven.test.skip=true -gs `pwd`/settings.xml clean package -U -Denv.trackerror=true'
                }
            }
        }

        stage('构建并推送镜像'){
           steps{
                container ('maven') {
                  sh 'docker build --build-arg SERVICE_SRC_PATH=$SERVICE_SRC_PATH \
                     --build-arg GENERATE_PATH=generated-resources/appassembler/jsw/$APP_NAME \
                     --build-arg RELEASE_NAME=$BRANCH_NAME-$BUILD_NUMBER \
                     --build-arg APP_NAME=$APP_NAME \
                     -f Dockerfile \
                     -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$BRANCH_NAME-$TAG_NAME-$BUILD_NUMBER-$COMMIT_ID \
                     --no-cache .'
                  withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) {
                        sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
                        sh 'docker push  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$BRANCH_NAME-$TAG_NAME-$BUILD_NUMBER-$COMMIT_ID'
                  }
                  sh 'docker tag  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$BRANCH_NAME-$TAG_NAME-$BUILD_NUMBER-$COMMIT_ID $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
                  sh 'docker push  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
                }
           }
        }
         stage("gitlab 打 tag"){
          when{
            expression{
              return params.TAG_NAME =~ /v.*/
            }
          }
         steps {
                withCredentials([usernamePassword(credentialsId: "$GITHUB_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
                                       sh 'git config --global user.email "[e-mail]" '
                                       sh 'git config --global user.name "xxxx" '
                                       sh 'git tag -a $TAG_NAME-$BUILD_NUMBER -m "$TAG_NAME" '
                                       sh 'git push https://$GIT_USERNAME:$GIT_PASSWORD@$DOWNLOAD_BASEDOMAIN/$PROGECT_GIT_PATH --tags --ipv4'
                                     }
                }
         }

        stage('部署测试环境') {
         // when{
         //   branch 'master'
         // }
          steps {
            //input(id: 'deploy-to-dev', message: 'deploy to dev?')
            kubernetesDeploy(configs: 'yaml/**', enableConfigSubstitution: true, kubeconfigId: "$KUBECONFIG_CREDENTIAL_ID_DEV")
          }
        }

        stage('部署生产环境') {
          when{
            expression{
              return params.TAG_NAME =~ /v.*/
            }
          }
          steps {
            input(id: 'deploy-to-prod', message: '是否允许发布生产?')
            kubernetesDeploy(configs: 'yaml/**', enableConfigSubstitution: true, kubeconfigId: "$KUBECONFIG_CREDENTIAL_ID_PROD")
          }
        }
    }
}

The Jenkinsfile describes the following processes:

• Download common template files (maven setting, yaml file for deployment, Dockerfile for building container images)
• Compiling Java applications with Maven
• Package the compiled Java application as a Docker image
• Push the built Docker image to the private DockerHub
• Deploy container images to various environments

The interface execution effect is:

2. Write the yaml file for deployment

 # java deployment基本配置
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: $APP_NAME
    component: ${PROJECT_NAMESPACE}-${APP_NAME}
    release: java-actuator-prometheus
    tier: backend
  name: ${PROJECT_NAMESPACE}-${APP_NAME}
  namespace: ${PROJECT_NAMESPACE}
spec:
  progressDeadlineSeconds: 600
  replicas: ${PODCOUNT}
  selector:
    matchLabels:
      app: $APP_NAME
      component: ${PROJECT_NAMESPACE}-${APP_NAME}
      tier: backend
  template:
    metadata:
      labels:
        app: $APP_NAME
        component: ${PROJECT_NAMESPACE}-${APP_NAME}
        tier: backend
        release: java-actuator-prometheus
      annotations:
        prometheus.io/path: /api/actuator/prometheus
        prometheus.io/port: '8090'
        prometheus.io/scrape: 'true'
    spec:
      volumes:
        - name: base-config
          configMap:
            name: base-config
            items:
              - key: server.properties
                path: server.properties
            defaultMode: 420
        - name: logconfig
          configMap:
            name: logconfig
            items:
              - key: logging-config.xml
                path: logging-config.xml
            defaultMode: 420
      containers:
        - env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
            - name: DEPLOY_ENV
              valueFrom:
                configMapKeyRef:
                  name: base-config
                  key: env
            - name: NACOS_SERVER
              valueFrom:
                configMapKeyRef:
                  name: base-config
                  key: nacos-server
            - name: DB_SERVER_ADDRESS
              valueFrom:
                configMapKeyRef:
                  name: base-config
                  key: DB_SERVER_ADDRESS
          image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$BRANCH_NAME-$TAG_NAME-$BUILD_NUMBER-$COMMIT_ID
          readinessProbe:
            httpGet:
              path: ${HEALTH_CHECK_URI}
              port: 8090
            initialDelaySeconds: 30
            timeoutSeconds: 10
            failureThreshold: 30
            periodSeconds: 5
          imagePullPolicy: Always
          name: ${PROJECT_NAMESPACE}-${APP_NAME}
          ports:
            - containerPort: 8080
              protocol: TCP
            - containerPort: 8090
              protocol: TCP
          resources:
            limits:
              cpu: 2000m
              memory: 600Mi
            requests:
              cpu: 1m
              memory: 100Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30


---
# 服务svc配置信息
apiVersion: v1
kind: Service
metadata:
  labels:
    app: ${APP_NAME}
    component: ${PROJECT_NAMESPACE}-${APP_NAME}
    release: java-actuator-prometheus
  name: ${PROJECT_NAMESPACE}-${APP_NAME}
  namespace: ${PROJECT_NAMESPACE}
  annotations:
    prometheus.io/path: /api/actuator/prometheus
    prometheus.io/port: '8090'
    prometheus.io/scrape: 'true'

spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
    - name: http-actuator
      port: 8090
      protocol: TCP
      targetPort: 8090
  selector:
    app: ${APP_NAME}
    component: ${PROJECT_NAMESPACE}-${APP_NAME}
    tier: backend
  sessionAffinity: None
  type: ClusterIP

Through the above yaml we will deploy the application load container and service SVC.

We described the following in the metadata of Deployment, which will be used later when deploying ServiceMonitor.

Deploy ServiceMonitor in Kubernetes

Prepare our ServiceMonitor deployment file corresponding to the Java service

 apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: java-actuator-prometheus
    component: java-actuator-prometheus
    heritage: Tiller
    release: prometh-java-actuator
  name: monitor-java-actuator-prometheus
  namespace: default
spec:
  endpoints:
    - honorLabels: true
      interval: 5s
      path: /api/actuator/prometheus
      port: http
  jobLabel: java-actuator-prometheus
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      release: java-actuator-prometheus

yaml describes what data we will collect under the namespace. Here we set the namespace to default, and we will collect the data under all the namespaces. At the same time, we set the release:xx under the selector to match the metadata of the microservices we deployed. If the release is consistent, then ServiceMonitor will collect the data of all services marked with release as java-actuator-prometheus under the namespace.

Deploy ServiceMonitor into the cluster

We can deploy it to the cluster via kubectl apply.

 kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: java-actuator-prometheus
    component: java-actuator-prometheus
    heritage: Tiller
    release: prometh-java-actuator
  name: monitor-java-actuator-prometheus
  namespace: default
spec:
  endpoints:
    - honorLabels: true
      interval: 5s
      path: /api/actuator/prometheus
      port: http
  jobLabel: java-actuator-prometheus
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      release: java-actuator-prometheus
EOF

After successful execution, we can search for ServiceMonitor under the CRD of the cluster and open the ServiceMonitor configuration where we can find our deployment. as the picture shows:

Of course, you can also query and verify through the command line.

Verify data collection and configure Grafana

View system Prometheus address and query data

We can find the Prometheus service integrated by the KubeSphere system at the following address in the cluster, as shown in the figure

Access the Prometheus web interface.

Through 3.1.1 we can see that the ip address of the Prometheus service is 172.17.107.29, and the default port is 9090. We enter http://172.17.107.29:9090 in the browser, and we can see as shown in the figure:

Configure custom monitoring and alerting in KubeSphere

1. Custom monitoring

We can access-cluster->monitoring alarm->custom monitoring to enter, as shown in the figure:

We click Create to see that KubeSphere has integrated some monitoring panels, here we choose Grafana.

After the next step, the system will let us upload the JSON template. We can download some general template configurations through the grafana official website . Here we use JVM (Micrometer) . On the right you can download the JSON file.

After importing, we can see the relevant monitoring indicators.

2. Custom Alerts

We can also use the system-integrated alarm policy to set custom alarm settings based on collected data. E.g:

Use external Grafana

1. Install Grafana

• Configure the application repository

  • In order to quickly install the Helm application, we can open the enterprise space - application management - application warehouse in turn;
  • Click the Add button on the right here we add the address of bitmap's application warehouse: https://charts.bitnami.com/bitnami ;
  • After adding, wait a moment for the application list to load.

• Install the grafana app

  • We open the enterprise space - project - click the specific project to be installed - click the application - click the create button on the right;
  • In the pop-up dialog box, click From Application Template, select the bitnami repository we just added from the application repository list, search for Grafana, click and install it.

2. Grafana data source

• We use the administrator account to log in to Grafana. The default password can be found in grafana-admin in the project's secret dictionary;
• After logging in, we click on the small gear-datasource on the left, select Add data source in the open page, then select Prometheus, and fill in the URL address of Prometheus we mentioned above in the URL. as the picture shows:
  
• Fill it out and drag it to the bottom, click save&test.

3. Import Dashbord

• We click ➕-import on the left side of the page;
• Enter the id of the corresponding template we obtained from the grafana official website and click load;
• In the next step, select the data source that Prometheus has configured for us and click import.

This article is published by OpenWrite , a multi-post blog platform!