Microservices have everything from code to k8s deployment (k8s deployment)

We use a series to explain the complete practice of microservices from requirements to online, from code to k8s deployment, from logging to monitoring, etc.

The whole project uses microservices developed by go-zero, which basically includes go-zero and some middleware developed by related go-zero authors. The technology stack used is basically the self-developed components of the go-zero project team, basically go -zero the whole family bucket.

Actual project address: https://github.com/Mikaelemmmm/go-zero-looklook

1 Overview

In the previous section, we have already built gitlab, jenkins, harbor, and k8s. In this section, we will write the pipline of jenkins and publish our services to k8s through jenkins.

2. Deploy middleware

Deploy mysql, redis, es, etc. outside of k8s, and simulate it as an online independent environment (as for online you want to deploy some middleware inside k8s, this will be handled by yourself, this time the focus is on how to develop go-zero Microservices are deployed inside the k8s cluster), here I directly use the docker-compose-env.yaml under the project, and install all the dependent third-party middleware environments directly on the srv-data.com (192.168.1.181) machine Server, provided that the server has docker and docker-compose installed.

Login to 192.168.1.181

 $ mkdir data && cd data && vim docker-compose.yml
$ docker-compose up -d
$ docker-compose ps #查看确认

3. Independent configuration

The configuration of each service is independent and placed in a git repository, so that only one person has the permission of the online repository. If the online configuration changes, directly modify the files of this repository. When jenkins does cd, it will first Pull the code and then pull the configuration of the corresponding service to build automatically. For details, see the pipline below.

[Q] Why not use the configuration center?

1) Modifying db, redis, etc. requires restarting services, but there are some configurations that do not require restarting services, and operation and maintenance must be remembered. It is easy to cause online accidents if you mix them up.

2) Easy to roll back. We released a new version online and changed the configuration of the new version. At this time, there is a problem with online user feedback. If you need to roll back quickly online, if we use the build file into the image, we can directly roll back the previous version of the code and configuration with one line of k8s command. If the configuration center is used, the code is rolled back, and the configuration of the previous version needs to be changed back to the configuration center, which is very troublesome.

The independent online warehouse directory structure is as follows (this structure is related to the writing in pipline)

Warehouse address: https://github.com/Mikaelemmmm/go-zero-looklook-pro-conf , just download it directly

1. Modify the middleware in the configuration. The database, redis, etc. must be changed to the machine 192.168.1.181. We regard this machine as the middleware of the online environment.

2. The other is our service discovery. We deploy it in k8s online, and go-zero directly supports k8s service discovery, so there is no need for etcd, etc. When configuring the zrpc client, we need to change it to target, k8s configuration method .

4. Write the pipline of jenkins

4.1 Configuration parameters

Visit http://192.168.1.180:8989/ to open jenkins, enter the jenkins home page, click on the left menu 新建Item

We first create identity the pipeline of the authorization service

Then click "General", select "This project is parameterized", "Add parameter", "Choice Parameter", as shown below

Then write the following

Save directly.

4.2 Write pipline

Scroll down to find Pipeline script , fill in the script content

 pipeline {
  agent any
  parameters {
      gitParameter name: 'branch', 
      type: 'PT_BRANCH',
      branchFilter: 'origin/(.*)',
      defaultValue: 'master',
      selectedValue: 'DEFAULT',
      sortMode: 'ASCENDING_SMART',
      description: '选择需要构建的分支'
  }

  stages {
      stage('服务信息')    {
          steps {
              sh 'echo 分支：$branch'
              sh 'echo 构建服务类型：${JOB_NAME}-$type'
          }
      }

      stage('拉取代码') {
          steps {
              checkout([$class: 'GitSCM', 
              branches: [[name: '$branch']],
              doGenerateSubmoduleConfigurations: false, 
              extensions: [], 
              submoduleCfg: [],
              userRemoteConfigs: [[credentialsId: 'gitlab-cert', url: 'ssh://git@192.168.1.180:2222/root/go-zero-looklook.git']]])
          }   
      }

      stage('获取commit_id') {
          steps {
              echo '获取commit_id'
              git credentialsId: 'gitlab-cert', url: 'ssh://git@192.168.1.180:2222/root/go-zero-looklook.git'
              script {
                  env.commit_id = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim()
              }
          }
      }

      stage('拉取配置文件') {
              steps {
                  checkout([$class: 'GitSCM', 
                  branches: [[name: '$branch']],
                  doGenerateSubmoduleConfigurations: false, 
                  extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'conf']], 
                  submoduleCfg: [],
                  userRemoteConfigs: [[credentialsId: 'gitlab-cert', url: 'ssh://git@192.168.1.180:2222/root/go-zero-looklook-pro-conf.git']]])
              }   
      }

      stage('goctl版本检测') {
          steps{
              sh '/usr/local/bin/goctl -v'
          }
      }
      
      stage('Dockerfile Build') {
          steps{
                 sh 'yes | cp  -rf conf/${JOB_NAME}/${type}/${JOB_NAME}.yaml  app/${JOB_NAME}/cmd/${type}/etc'   //线上配置文件
                 sh 'cd app/${JOB_NAME}/cmd/${type} && /usr/local/bin/goctl docker -go ${JOB_NAME}.go && ls -l'
                 script{
                     env.image = sh(returnStdout: true, script: 'echo ${JOB_NAME}-${type}:${commit_id}').trim()
                 }
                 sh 'echo 镜像名称：${image} && cp app/${JOB_NAME}/cmd/${type}/Dockerfile ./  && ls -l && docker build  -t ${image} .'
          }
      }

      stage('上传到镜像仓库') {
          steps{
                //docker login 这里要注意，会把账号密码输出到jenkins页面，可以通过port.sh类似方式处理，官网文档有这里我就不详细写了
              sh 'docker login --username=${docker_username} --password=${docker_pwd} http://${docker_repo}' 
              sh 'docker tag  ${image} ${docker_repo}/go-zero-looklook/${image}'
              sh 'docker push ${docker_repo}/go-zero-looklook/${image}'
          }
      }

      stage('部署到k8s') {
          steps{
              script{
                  env.deployYaml = sh(returnStdout: true, script: 'echo ${JOB_NAME}-${type}-deploy.yaml').trim()
                  env.port=sh(returnStdout: true, script: '/root/port.sh ${JOB_NAME}-${type}').trim()
              }

              sh 'echo ${port}'

              sh 'rm -f ${deployYaml}'
              sh '/usr/local/bin/goctl kube deploy -secret docker-login -replicas 2 -nodePort 3${port} -requestCpu 200 -requestMem 50 -limitCpu 300 -limitMem 100 -name ${JOB_NAME}-${type} -namespace go-zero-looklook -image ${docker_repo}/${image} -o ${deployYaml} -port ${port} --home /root/template'
              sh '/usr/local/bin/kubectl apply -f ${deployYaml}'
          }
      }

       stage('Clean') {
           steps{
               sh 'docker rmi -f ${image}'
               sh 'docker rmi -f ${docker_repo}/${image}'
               cleanWs notFailBuild: true
           }
       }
  }
}

Very important! ! !

1. Build optimization: When dockerfile is generated in pipline, we use k8s to deploy without etcd, but this way of deployment requires a specified account (with the permission to get in the endpoints of k8s, just use the default default, create one each time The new namespace k8s will automatically create a default for us), but the k8s yml generated by goctl does not add the specified account option. This has been feedback and may be added in subsequent versions. Here we also use the template, the same template It is in the project directory https://github.com/Mikaelemmmm/go-zero-looklook/tree/main/deploy/goctl , you can build and specify this template in pipline
2. ${credentialsId} should be replaced with your specific credential value, that is, a string of strings in the [Add Credentials] module. We previously configured gitlab-cert, so fill in gitlab-cert here. If you are not this, you need to replace it yourself. ${gitUrl} needs to be replaced with the git repository address of your code, and other variables in the form of ${xxx} do not need to be modified, just keep them as they are.
3. This is a little different from the official documentation. Because of the different directory of my project folder, I manually adjusted the dockerfile generated by goctl. In a dockerfile that I did not generate during construction, I put the dockerfile together when I created the project. Put it in the directory, so you don't need goctl when building the image

5. Configure k8s to pull private warehouse images

By default, k8s can only pull the public image of the harbor mirror warehouse. If you pull the private warehouse mirror, it will report ErrImagePull and ImagePullBackOff errors

1. First log in harbor on the jenkins release machine

 $ docker login 192.168.1.180:8077
$ Username: admin
$ Password:
Login Succeeded

2. Generate the login harbor configuration file in k8s

 #查看上一步登陆harbor生成的凭证
$ cat /root/.docker/config.json  
{
    "auths": {
        "192.168.1.180:8077": {
            "auth": "YWRtaW46SGFyYm9yMTIzNDU="
        }
}

3. Encrypt the key file with base64

 $ cat /root/.docker/config.json  | base64 -w 0

ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEuMTgwOjgwNzciOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=

4. Create docker-secret.yaml

 apiVersion: v1
kind: Secret
metadata:
  name: docker-login
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEuMTgwOjgwNzciOiB7CgkJCSJhdXRoIjogIllXUnRhVzQ2U0dGeVltOXlNVEl6TkRVPSIKCQl9Cgl9Cn0=

 $ kubectl create -f docker-secret.yaml -n go-zero-looklook

secret "docker-login" created

6. Build

We enter the home page, click on identity to enter the details page

Then you can see that the identity service we configured above, as shown in the figure below, click "Build with Parameters", then select rpc, and click "Start Build"

The first build will fail when pulling the code, it should be initializing something, just click it again.

Deployment was successful

In the same way, go to build identity-api, then configure usercenter service to build usercenter-rpc, build usercenter-api, and then configure other services and build, this time we only build identity-api, identity-rpc, usercenter-rpc , usercenter-api to demonstrate to you.

6. Add gateway

Because our api service is published in k8s through goctl, the nodeport port will be exposed. For the index, we will look at the nodeport port service of the service under the go-zero-looklook namespace in k8s, and then configure the nodeport in nignx.

This time, we installed nginx on a separate virtual machine outside k8s, exposed the port of the k8s backend api service to nginx through nodeport, and then nginx configures this api service in the configuration, so that nginx acts as a gateway.

The installation of nginx will not be discussed here. Remember to have the auth_request module. If not, install it yourself.

nginx configuration

 server{
    listen 8081;
    access_log /var/log/nginx/looklook.com_access.log;
    error_log /var/log/nginx//looklook.com_error.log;

    location /auth {
        internal;
      proxy_set_header X-Original-URI $request_uri;
        proxy_pass_request_body off;
        proxy_set_header Content-Length "";
        proxy_pass http://192.168.1.182:31001/identity/v1/verify/token;
    }

    location ~ /usercenter/ {
       auth_request /auth;
       auth_request_set $user $upstream_http_x_user;
       proxy_set_header x-user $user;

       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header REMOTE-HOST $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_pass http://192.168.1.182:31002;
   }

   location ~ /travel/ {
       auth_request /auth;
       auth_request_set $user $upstream_http_x_user;
       proxy_set_header x-user $user;

       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header REMOTE-HOST $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_pass http://192.168.1.182:31003;
   }

   location ~ /order/ {
       auth_request /auth;
       auth_request_set $user $upstream_http_x_user;
       proxy_set_header x-user $user;

       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header REMOTE-HOST $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_pass http://192.168.1.182:31004;
   }

   location ~ /payment/ {
       auth_request /auth;
       auth_request_set $user $upstream_http_x_user;
       proxy_set_header x-user $user;

       proxy_set_header Host $http_host;
       proxy_set_header X-Real-IP $remote_addr;
       proxy_set_header REMOTE-HOST $remote_addr;
       proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_pass http://192.168.1.182:31005;
   }
}

If it is online, you should configure multiple nignx to maintain high availability, there will be a slb in front of nignx, your domain name including https configuration should be resolved to slb, there is a firewall in front of slb and so on.

8. Conclusion

At this point, the whole series is over. The overall architecture diagram should be as shown in the first article. I hope this series can help you.

project address

https://github.com/zeromicro/go-zero

Welcome go-zero and star support us!

WeChat exchange group

Follow the official account of " Microservice Practice " and click on the exchange group to get the QR code of the community group.