Kubernetes + Spring Cloud integrated link tracking SkyWalking

I. Overview
1. What is SkyWalking?
Application performance monitoring tools for distributed systems, designed for microservices, cloud-native architectures, and container-based (Docker, K8s, Mesos) architectures. Provide integrated solutions for distributed tracking, service grid telemetry analysis, measurement aggregation and visualization.

Official website address: http://skywalking.apache.org/

2. SkyWalking features
Multiple monitoring methods, language probes and Service Mesh
Multi-language automatic probe, Java, .NET Core and Node.JS
Lightweight and efficient, no need for big data
Modular, multiple mechanisms for UI, storage, and cluster management are optional
Support alarm
Excellent visualization solution
3. Overall structure

The entire structure is divided into four parts: top, bottom, left, and right:

In consideration of making the description simpler, we discard the Metric indicator correlation and focus on the Tracing link-related functions.

The upper part of the Agent: is responsible for collecting link information from the application and sending it to the SkyWalking OAP server. Currently it supports Tracing data information provided by SkyWalking, Zikpin, Jaeger, etc. What we currently use is that SkyWalking Agent collects SkyWalking Tracing data and transmits it to the server.
The next part of SkyWalking OAP: Responsible for receiving the Tracing data information sent by the Agent, then performing analysis (Analysis Core), storing it in the external storage (Storage), and finally providing the query (Query) function.

Storage on the right: Tracing data storage. Currently, it supports multiple storages of ES, MySQL, Sharding Sphere, TiDB, and H2. And we are currently using ES, the main consideration is the SkyWalking development team’s own production

The left part of SkyWalking UI: Responsible for providing console, checking links, etc.
The simple overview principle is as follows:
![uploading...]()

Second, build skywalking
1. Environmental preparation
Mkubernetes version: 1.18.5
Nginx Ingress version: 2.2.8
Helm version: 3.2.4
Persistent storage driver: NFS
2. Deploy using chart
This article mainly describes how to use Helm Charts to deploy SkyWalking to a Kubernetes cluster. For related documents, please refer to skywalking-kubernetes

Four methods currently recommended:

Use the helm serve provided by helm 3 to start the local helm repo
Use local chart file deployment
Use the repo function provided by harbor
Deploy directly from the official repo (not satisfied yet)
❝
Note: At present, the chart of skywalking has not been submitted to the official warehouse, please refer to the first three methods for deployment

❞
2.1. Download the chart file
You can directly use the local file to deploy skywalking. After downloading the skywalking chart according to the above steps, directly use the following command to deploy:

git clone https://github.com/apache/skywalking-kubernetes
cd skywalking-kubernetes/chart
helm repo add elastic https://helm.elastic.co
helm dep up skywalking
export SKYWALKING_RELEASE_NAME=skywalking # define your own name
export SKYWALKING_RELEASE_NAMESPACE=default # Define your own namespace
2.2, define the existing es parameter file
Modify values-my-es.yaml:

oap:
image:

tag: 8.1.0-es7      # Set the right tag according to the existing Elasticsearch version

storageType: elasticsearch7

ui:
image:

tag: 8.1.0

elasticsearch:
enabled: false
config: # For users of an existing elasticsearch cluster,takes effect when elasticsearch.enabled is false

host: elasticsearch-client
port:
  http: 9200
user: "elastic"         # [optional]
password: "admin@123"     # [optional]

2.3, helm installation
helm install "${SKYWALKING_RELEASE_NAME}" skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" \
-f ./skywalking/values-my-es.yaml
After the installation is complete, we verify the installation situation:

$ kubectl get deployment -n skywalking
NAME READY UP-TO-DATE AVAILABLE AGE
my-skywalking-oap 2/2 2 2 9m
my-skywalking-ui 1/1 1 1 9m
3. Use Skywalking Agent
Using agent in Java provides the following three ways for you to choose

Use the official basic image
Build the agent package into the existing base image
Mount agent in sidecar mode (recommended)
1. Use the official basic image
To view the basic image provided by the official docker hub, you only need to build the service image from this image, which is more convenient to integrate directly into Jenkins

2. Build the agent package into the existing base image
The reason for providing this method is: the official mirror is a streamlined mirror, and it is openjdk. There may not be many commands, and you need to install it twice, so skip it here.

3. Mount agent in sidecar mode
Since the service is deployed in Kubernetes, Skywalking Agent is used in this way. The advantage of this method is that there is no need to modify the original basic image or rebuild a new service image. Instead, it uses the sidecar mode to share the volume. Mount the relevant files required by the agent to the existing service image.

3.1, build skywalking agent image
Build it yourself, refer to: https://hub.docker.com/r/prophet/skywalking-agent

Build with the following dockerfile:

FROM alpine:3.8

LABEL maintainer="[e-mail]"

ENV SKYWALKING_VERSION=8.1.0

ADD http://mirrors.tuna.tsinghua.edu.cn/apache/skywalking/${SKYWALKING_VERSION}/apache-skywalking-apm-${SKYWALKING_VERSION}.tar.gz /

RUN tar -zxvf /apache-skywalking-apm-${SKYWALKING_VERSION}.tar.gz && \

mv apache-skywalking-apm-bin skywalking && \
mv /skywalking/agent/optional-plugins/apm-trace-ignore-plugin* /skywalking/agent/plugins/ && \
echo -e "\n# Ignore Path" >> /skywalking/agent/config/agent.config && \
echo "# see https://github.com/apache/skywalking/blob/v8.1.0/docs/en/setup/service-agent/java-agent/agent-optional-plugins/trace-ignore-plugin.md" >> /skywalking/agent/config/agent.config && \
echo 'trace.ignore_path=${SW_IGNORE_PATH:/health}' >> /skywalking/agent/config/agent.config

docker build -t 172.16.106.237/monitor/skywalking-agent:8.1.0 .
After the docker build is completed, push to the warehouse.

3.2, mount using sidecar
The sample configuration file is as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-skywalking
spec:
replicas: 1
selector:

matchLabels:
  app: demo-skywalking

strategy:

rollingUpdate:
  maxSurge: 1
  maxUnavailable: 0
type: RollingUpdate

template:

metadata:
  labels:
    app: demo-skywalking
spec:
  initContainers:
    - name: init-skywalking-agent
      image: 172.16.106.237/monitor/skywalking-agent:8.1.0
      command:
        - 'sh'
        - '-c'
        - 'set -ex;mkdir -p /vmskywalking/agent;cp -r /skywalking/agent/* /vmskywalking/agent;'
      volumeMounts:
        - mountPath: /vmskywalking/agent
          name: skywalking-agent
  containers:
    - image: nginx:1.7.9
      imagePullPolicy: Always
      name: nginx
      ports:
        - containerPort: 80
          protocol: TCP
      volumeMounts:
        - mountPath: /opt/skywalking/agent
          name: skywalking-agent
  volumes:
    - name: skywalking-agent
      emptyDir: {}

The above is the deployment.yaml file for mounting sidecar. Taking nginx as a service as an example, the agent is mainly mounted by sharing the volume. First, initContainers mounts /vmskywalking/agent in sw-agent-sidecar through the skywalking-agent volume. , And add the agent directory cp in the image built above to the /vmskywalking/agent directory. After the completion, the skywalking-agent volume is also mounted when nginx starts, and it is mounted to the /opt/skywalking/agent directory of the container, This completes the sharing process.

Fourth, transform the Spring Cloud application
1. Docker packaged and pushed to the warehouse
Modify the dockerfile configuration to integrate skywalking agent:

FROM insideo/centos7-java8-build
VOLUME /tmp
ADD mall-admin.jar app.jar
RUN bash -c 'touch /app.jar'
RUN ln -snf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo Asia/Shanghai > /etc/timezone
ENTRYPOINT ["java","-Dapp.id=svc-mall-admin","-javaagent:/opt/skywalking/agent/skywalking-agent.jar","-Dskywalking.agent.service_name=svc-mall-admin","-Dskywalking.collector.backend_service=my-skywalking-oap.skywalking.svc.cluster.local:11800","-jar","-Dspring.profiles.active=prod","-Djava.security.egd=file:/dev/./urandom","/app.jar"]
After the change, run maven package directly to package this project into a mirror image.

note:

❝
When k8s creates a Service, it creates a corresponding DNS entry. The format of this entry is <service-name>.<namespace-name>.svc.cluster.local, which means that if the container only uses <service-name>, it will be resolved as a local service to the namespace. If you want to access across namespaces, you need to use a fully qualified domain name.

❞
2. Write the deployment script of the yaml version of k8s
Here I take one of the services as an example:

---

apiVersion: apps/v1
kind: Deployment
metadata:
name: svc-mall-admin
spec:
replicas: 1
selector:

matchLabels:
  app: svc-mall-admin

strategy:

rollingUpdate:
  maxSurge: 1
  maxUnavailable: 0
type: RollingUpdate

template:

metadata:
  labels:
    app: svc-mall-admin
spec:
  initContainers:
    - name: init-skywalking-agent
      image: 172.16.106.237/monitor/skywalking-agent:8.1.0
      command:
        - 'sh'
        - '-c'
        - 'set -ex;mkdir -p /vmskywalking/agent;cp -r /skywalking/agent/* /vmskywalking/agent;'
      volumeMounts:
        - mountPath: /vmskywalking/agent
          name: skywalking-agent
  containers:
    - image: 172.16.106.237/mall_repo/mall-admin:1.0
      imagePullPolicy: Always
      name: mall-admin
      ports:
        - containerPort: 8180
          protocol: TCP
      volumeMounts:
        - mountPath: /opt/skywalking/agent
          name: skywalking-agent
  volumes:
    - name: skywalking-agent
      emptyDir: {}

---

apiVersion: v1
kind: Service
metadata:
name: svc-mall-admin
spec:
ports:

- name: http
  port: 8180
  protocol: TCP
  targetPort: 8180

selector:

app: svc-mall-admin

Then you can run it directly, and it can run all the projects.

Five, test verification
When you are done, you can go to the SkyWalking UI to check whether the link collection is successful.

1. Test application API
First, request the API provided by the Spring Cloud application. Because we want to track down the link.

2. View the SkyWalking UI interface

Insert picture description here
Here, we will see three very important concepts in SkyWalking:

Service (Service): Represents a series or set of workloads that provide the same behavior to the request. When using Agent or SDK, you can define the name of the service. If not defined, SkyWalking will use the name you defined on the platform (for example, Istio). Here, we can see that the service of the Spring Cloud application is svc-mall-admin, which is defined in the agent environment variable service_name.

Service Instance: Each workload in the above-mentioned set of workloads is called an instance. Just like pods in Kubernetes, a service instance is not necessarily a process on the operating system. But when you are using Agent, a service instance is actually a real process on the operating system. Here, we can see that the service of the Spring Cloud application is UUID@hostname, which is automatically generated by the Agent.

Endpoint: The request path received by a specific service, such as HTTP URI path and gRPC service class name + method signature.

Here, we can see an endpoint of the Spring Cloud application, which is the API interface /mall-admin/admin/login.

For more agent parameter introduction, refer to: 160dd44da5d658 https://github.com/apache/skywalking/blob/v8.1.0/docs/en/setup/service-agent/java-agent/README.md

Click the "Topology" menu to enter the interface for viewing the topology:
Click the "Tracking" menu to enter the interface for viewing link data:

Six, summary
This article describes in detail how to use Kubernetes + Spring Cloud to integrate SkyWalking. By the way, call chain monitoring is an indispensable component in the current microservice system. Distributed tracking, service grid telemetry analysis, measurement aggregation and visualization are still quite good. Yes, here we chose Skywalking, the specific reasons and details of the gameplay are not detailed here.