k8s deploys skywalking and connects pod applications to link tracking

I. Overview

I wrote two articles earlier about using docker to deploy spring boot and tomcat projects, and connecting them to skywalking. This article mainly introduces the use of k8s to deploy skywalking and connecting pod applications to link tracking.

Second, use helm to deploy skywalking

The prerequisite for using helm in k8s is to install the helm client first. For the installation of helm, you can check the official documentation.

Install helm official document address: https://helm.sh/docs/intro/install/

Here are two ways to deploy skywalking, one is to use Artifact Hub , and the other is to deploy through the source files provided by GitHub. The essence of these two deployment methods is the same, the difference is that the usual update on GitHub is usually before Artifact Hub .

1. Use the chart provided by Artifact Hub

This is Artifact Hub , search skywalking, you can see the following figure:

As shown in the figure below, click INSTALL to view and add the helm warehouse:

The command to add skywalking chart warehouse is as follows:

helm repo add choerodon https://openchart.choerodon.com.cn/choerodon/c7n

Use the following command to view the added repo, here is a list of other repositories I added:

helm repo list
NAME                    URL                                               
oteemocharts            https://oteemo.github.io/charts                   
kubeview                https://benc-uk.github.io/kubeview/charts         
oteemo-charts           https://oteemo.github.io/charts                   
jenkinsci               https://charts.jenkins.io/                        
ygqygq2                 https://ygqygq2.github.io/charts/                 
prometheus-community    https://prometheus-community.github.io/helm-charts
my-chart                https://wangedison.github.io/k8s-helm-chart/      
carlosjgp               https://carlosjgp.github.io/open-charts/          
choerodon               https://openchart.choerodon.com.cn/choerodon/c7n  

Use the following command to search for skywalking in the warehouse:

helm search repo skywalking
NAME                        CHART VERSION    APP VERSION    DESCRIPTION                 
choerodon/skywalking        6.6.0            6.6.0          Apache SkyWalking APM System
choerodon/skywalking-oap    0.1.3            0.1.3          skywalking-oap for Choerodon
choerodon/skywalking-ui     0.1.4            0.1.4          skywalking-ui for Choerodon 
choerodon/chart-test        1.0.0            1.0.0          skywalking-ui for Choerodon 

In order to do some custom configuration, such as modifying the version number, use the following command to download the chart to the local:

# 下载 chart 到本地
[root@k8s-node01 chart-test]# helm pull choerodon/skywalking
# 查看下载的内容
[root@k8s-node01 chart-test]# ll
total 12
-rw-r--r-- 1 root root 10341 Apr 21 11:12 skywalking-6.6.0.tgz
# 解压 chart 包
[root@k8s-node01 chart-test]# tar -zxvf skywalking-6.6.0.tgz 
skywalking/Chart.yaml
skywalking/values.yaml
skywalking/templates/_helpers.tpl
skywalking/templates/istio-adapter/adapter.yaml
skywalking/templates/istio-adapter/handler.yaml
skywalking/templates/istio-adapter/instance.yaml
skywalking/templates/istio-adapter/rule.yaml
skywalking/templates/mysql-init.job.yaml
skywalking/templates/oap-clusterrole.yaml
skywalking/templates/oap-clusterrolebinding.yaml
skywalking/templates/oap-deployment.yaml
skywalking/templates/oap-role.yaml
skywalking/templates/oap-rolebinding.yaml
skywalking/templates/oap-serviceaccount.yaml
skywalking/templates/oap-svc.yaml
skywalking/templates/ui-deployment.yaml
skywalking/templates/ui-ingress.yaml
skywalking/templates/ui-svc.yaml
skywalking/.auto_devops.sh
skywalking/.choerodon/.docker/config.json
skywalking/.gitlab-ci.yml
skywalking/.helmignore
skywalking/Dockerfile
skywalking/README.md

To customize the configuration, you can modify the values.yaml file:

vim skywalking/values.yaml

As you can see, the skywalking here uses the mysql database by default, and the version of skywalking used is 6.6.0. You can modify the corresponding version number according to your needs. After the modification is completed, save and exit. For more configuration modification instructions, please 16081816a4bc9c skywalking official chart instructions .

You can use the following command to install the chart and specify a custom configuration file:

# Usage:  helm install [NAME] [CHART] [flags]
helm install  skywalking skywaling -f skywalking/values.yaml

Using the above command will install skywalking in the default namespace, we can use the -n namespace parameter to specify a specific namespace, provided that this namespace must exist.

After the installation is successful, you can use the following command to view the installed chart. The installed chart is called release:

helm list

You can use the following command to uninstall the chart:

# Usage:  helm uninstall RELEASE_NAME [...] [flags]
helm uninstall skywalking -n default

2. Download the chart source file from GitHub to deploy skywalking

The GitHub address of the skywalking chart is: https://github.com/apache/skywalking-kubernetes

Use the following command to install git in a Linux system:

yum install -y git

Use the following command to clone the code:

git clone https://github.com/apache/skywalking-kubernetes

Go to the chart directory of skywalking, you can see the following content:

[root@k8s-node01 chart-test]# cd skywalking-kubernetes/chart/skywalking/
[root@k8s-node01 skywalking]# ll
total 84
-rw-r--r-- 1 root root  1382 Apr 21 11:35 Chart.yaml
drwxr-xr-x 3 root root  4096 Apr 21 11:35 files
-rw-r--r-- 1 root root   877 Apr 21 11:35 OWNERS
-rw-r--r-- 1 root root 42593 Apr 21 11:35 README.md
drwxr-xr-x 3 root root  4096 Apr 21 11:35 templates
-rw-r--r-- 1 root root  1030 Apr 21 11:35 values-es6.yaml
-rw-r--r-- 1 root root  1031 Apr 21 11:35 values-es7.yaml
-rw-r--r-- 1 root root  1366 Apr 21 11:35 values-my-es.yaml
-rw-r--r-- 1 root root 10184 Apr 21 11:35 values.yaml

You can see that the author has very intimately defined three custom configuration files for us: values-es6.yaml , values-es7.yaml and values-my-es.yaml , corresponding to the configuration using es6, es7 and external es storage respectively. Because I am using an external own es cluster here, and the version of es is 6.7.0, I need to modify the values-my-es.yaml file as follows:

# Default values for skywalking.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

oap:
  image:
    tag: 8.5.0-es6      # Set the right tag according to the existing Elasticsearch version
  storageType: elasticsearch # elasticsearch 对应 es6 ，elasticsearch7 对应 es7

ui:
  image:
    tag: 8.5.0

elasticsearch:
  enabled: false # 由于使用 外部的 es，所以这里需要设置为 false，因为设置为 true 会在 k8s 中部署 es
  config:               # For users of an existing elasticsearch cluster,takes effect when `elasticsearch.enabled` is false
    host: your.elasticsearch.host.or.ip
    port:
      http: 9200
    user: "xxx"         # [optional]
    password: "xxx"     # [optional]

You can also modify the values.yaml file, such as opening ingress. For more detailed configuration, you can check skywalking-kubernetes in GitHub. After configuring it according to your needs, you can use the following command to install the chart:

helm install "${SKYWALKING_RELEASE_NAME}" skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}"  -f ./skywalking/values-my-es.yaml

After the installation is complete, you can use the following command to check whether the pod starts normally:

[root@k8s-node01 ~]# kubectl get pod -n default
NAME                                     READY   STATUS      RESTARTS   AGE
skywalking-es-init-v6sbn                 0/1     Completed   0          1h
skywalking-oap-5c4d5bf887-4cvjk          1/1     Running     0          1h
skywalking-oap-5c4d5bf887-g75fj          1/1     Running     0          1h
skywalking-ui-6cd4bbd858-sbpvt           1/1     Running     0          1h

2. Use sidecar to connect pod to link tracking

Earlier we briefly introduced the use of helm to deploy skywalking, and the following describes how to use sidecar to connect pod to link tracking. Java microservices access to skywalking can use SkyWalking Java Agent to report monitoring data, which requires the java -javaagent:<skywalking-agent-path> specify the skywalking agent probe package through 06068816a4c070 in the startup parameters. There are usually three ways to integrate:

• Use the official basic image skywalking-base ;
• Build the agent package into the existing image;
• Mount the agent through the sidecar mode;

The first two methods are briefly introduced in the previous article. Here we mainly introduce how to use sidecar to connect pod to link tracking. This method does not need to modify the original basic image, nor does it need to rebuild a new service image. In the sidecar mode, the relevant files needed by the agent will be directly mounted to the existing service image through the shared volume. The principle of the sidecar mode is very simple. It is to deploy an initial container in the pod. The function of this initial container is to share the skywalking agent with the application container in the pod.

1. What is init container

Init Container is the container used for initialization. It can be one or more. If there are more than one, these containers will be executed in a defined order. Only after all Init Containers are executed, the main container will be started. We know that all containers in a Pod share the data volume and network namespace, so the data generated in the Init Container can be used by the main container.

2. Customize the skywalking agent image

Before starting to connect a java microservice to skywalking by sidecar, we need to build a public image of skywalking agent. The specific steps are as follows:

• Use the following command to download and unzip the skywalking agent:

# 下载 skywalking-8.5.0 for es6 版本的发布包，与部署的 skywalking 后端版本一致
wget https://www.apache.org/dyn/closer.cgi/skywalking/8.5.0/apache-skywalking-apm-8.5.0.tar.gz
# 将下载的发布包解压到当前目录
tar -zxvf apache-skywalking-apm-8.5.0.tar.gz

• Write the Dockerfile file in the same level directory of the skywalking distribution package decompressed in the previous step. The specific content is as follows:

FROM busybox:latest
LABEL maintainer="xiniao"
COPY apache-skywalking-apm-bin/agent/ /usr/skywalking/agent/

The basic image used in the above Dockefile file is the bosybox image, not the SkyWalking release image. This ensures that the built sidecar image is kept to a minimum.

• Use the following command to build the image:

docker build -t skywalking-agent-sidecar:8.5.0 .

Use the following command to view the built image:

docker images |grep agent
skywalking-agent-sidecar          8.5.0             98290e961b49        5 days ago          32.6MB

• Use the following command to tag the image and push it to my Alibaba Cloud image warehouse:

docker tag skywalking-agent-sidecar:8.5.0 registry.cn-shenzhen.aliyuncs.com/devan/skywalking-agent-sidecar:8.5.0

Use the following command to push the image to the remote warehouse:

docker push registry.cn-shenzhen.aliyuncs.com/devan/skywalking-agent-sidecar:8.5.0

3. Access to skywalking in sidecar mode

Above we built the public Docker image of SkyWalking Java Agent by hand-building. Next, we will demonstrate how to publish the Java service to the K8s cluster by writing the Kubernetes service release file to automatically integrate the Agent in the form of SideCar. Enter the SkyWalking service.

• Create a deploy-skywalking.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-boot-skywalking-demo
  namespace: default
  labels:
    app: spring-boot-skywalking-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spring-boot-skywalking-demo
  template:
    metadata:
      labels:
        app: spring-boot-skywalking-demo
    spec:
      #构建初始化镜像(通过初始化镜像的方式集成SkyWalking Agent)
      initContainers:
        - image: registry.cn-shenzhen.aliyuncs.com/devan/skywalking-agent-sidecar:8.5.0
          name: sw-agent-sidecar
          imagePullPolicy: IfNotPresent
          command: [ "sh" ]
          args:
            [
                "-c",
                "cp -R /usr/skywalking/agent/* /skywalking/agent",
            ]
          volumeMounts:
            - mountPath: /skywalking/agent
              name: sw-agent
      containers:
        - name: spring-boot-skywalking-demo
          image: ${ORIGIN_REPO}/spring-boot-skywalking-demo:${IMAGE_TAG}
          imagePullPolicy: Always
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: BUILD_TAG
              value: ${BUILD_TAG}
            - name: NAMESPACE
              value: default
            #这里通过JAVA_TOOL_OPTIONS，而不是JAVA_OPTS可以实现不通过将agent命令加入到java应用jvm参数而实现agent的集成
            - name: JAVA_TOOL_OPTIONS
              value: -javaagent:/usr/skywalking/agent/skywalking-agent.jar
            - name: SW_AGENT_NAME
              value: spring-boot-skywalking-demo
            - name: SW_AGENT_COLLECTOR_BACKEND_SERVICES
              # FQDN: servicename.namespacename.svc.cluster.local
              value: skywalking-oap.default.svc:11800
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 200m
              memory: 500Mi
          volumeMounts:
            - mountPath: /usr/skywalking/agent
              name: sw-agent
      volumes:
        - name: sw-agent
          emptyDir: { }


---
apiVersion: v1
kind: Service
metadata:
  name: spring-boot-skywalking-demo
  namespace: default
  labels:
    app: spring-boot-skywalking-demo
spec:
  ports:
    - name: port
      port: 80
      protocol: TCP
      targetPort: 8080
  selector:
    app: spring-boot-skywalking-demo
  type: ClusterIP

spec.volumes refers to the volume in the pod, and spec.containers.volumeMounts the specified volume to the specified location of the container, which is equivalent to Host directory: container directory, we use emptyDir{} here, which is equivalent to a share Volume is a temporary directory, and its life cycle is equivalent to that of a Pod. The command executed by the initial container is sh -c cp -R /usr/skywalking/agent/* /skywalking/agent , which means that the skywalking agent is copied to the shared directory. The main container is associated with the shared directory, so the main container can access the skywalking agent.

Use the following command to deploy the application:

 kubectl apply -f deploy-skywalking.yaml

to sum up

This article briefly introduces the use of helm to deploy skywalking, about the use of helm and how to customize the chart, you can write an article to introduce it later, if you want to learn more, it is recommended to check the official document. There is also a brief introduction of pod applications accessing the SkyWalking service in SideCar mode, mainly to understand the role of the initial container initContainers. The initial container is executed before the main container is started, and can share the data volume with the main container to share the network namespace.

Reference article
microservice access SkyWalking, three minutes to teach you how to play!