The most powerful performance monitoring system! Combine powerful functions in one

SkyWalking is an application performance monitoring system, specially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures. In addition to application indicator monitoring, it can also track distributed call links. Components with similar functions include: Zipkin, Pinpoint, CAT, etc.

Take a few pictures to see the effect, and then build and use it step by step:

Concept and architecture

SkyWalking is an open source monitoring platform used to collect, analyze, aggregate, and visualize data from services and cloud native infrastructure. SkyWalking provides an easy way to maintain a clear view of distributed systems, even across clouds. It is a modern APM designed specifically for cloud-native, container-based distributed systems.

SkyWalking monitors applications from three dimensions: service (service), service instance (instance), endpoint (endpoint)

Not much to say about services and instances, the endpoint is a path or URI in the service

SkyWalking allows users to understand the topology relationship between Services and Endpoints, to view the metrics of every Service/Service Instance/Endpoint and to set alarm rules.

SkyWalking allows users to understand the topological relationship between services and endpoints, view the metrics of each service/service instance/endpoint, and set alert rules.

Architecture

SkyWalking is logically divided into four parts: Probes, Platform backend, Storage, UI.

This structure is very clear. The probe is the Agent responsible for collecting data and reporting it to the server, the server processes and stores the data, and the UI is responsible for display.

download and install

There are two versions of SkyWalking, ES version and non-ES version. If we decide to use ElasticSearch as storage, then download the es version.

• https://skywalking.apache.org/downloads
• https://archive.apache.org/dist/skywalking/

• The agent directory will be copied to the machine where each service is used as a probe in the future
• The bin directory is the service startup script
• The config directory is the configuration file
• The oap-libs directory is the jar package required for the operation of the oap service
• The webapp directory is the jar package required for the operation of the web service

Next, you have to select the storage. The supported storages are:

• H2
• ElasticSearch 6, 7
• MySQL
• TiDB
• InfluxDB

As a monitoring system, first exclude H2 and MySQL. InfluxDB is recommended here. It is a time series database and is very suitable for this scenario.

But I’m not very familiar with InfluxDB, so I’ll use ElasticSearch7 first.

https://github.com/apache/skywalking/blob/master/docs/en/setup/backend/backend-storage.md

install ElasticSearch

https://www.elastic.co/guide/en/elasticsearch/reference/7.10/targz.html

# 启动
./bin/elasticsearch -d -p pid
# 停止
pkill -F pid

ElasticSearch7.x requires Java 11 or above, but if you set the environment variable JAVA_HOME, it will use your own Java version

Usually, the following three errors will be reported during the startup process:

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[3]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

Solution: Add the following content to the /etc/security/limits.conf file:

* soft nofile 65536
* hard nofile 65536
* soft nproc  4096
* hard nproc  4096

The modification results can be viewed through the following four commands:

ulimit -Hn
ulimit -Sn
ulimit -Hu
ulimit -Su

Modify the /etc/sysctl.conf file and append the following content:

vm.max_map_count=262144

Modify the es configuration file elasticsearch.yml, uncomment, and reserve a node

cluster.initial_master_nodes: ["node-1"]

In order to be able to access in ip:port mode, the network configuration needs to be modified

network.host: 0.0.0.0

After modification, it looks like this:

At this point, ElasticSearch has been successfully started, and one node is not enough. Here we use three nodes to build a cluster

• 192.168.100.14 config/elasticsearch.yml

cluster.name: my-monitor
node.name: node-1
network.host: 192.168.100.14
http.port: 9200
discovery.seed_hosts: ["192.168.100.14:9300", "192.168.100.15:9300", "192.168.100.19:9300"]
cluster.initial_master_nodes: ["node-1"]
192.168.100.15  config/elasticsearch.yml

cluster.name: my-monitor
node.name: node-2
network.host: 192.168.100.15
http.port: 9200
discovery.seed_hosts: ["192.168.100.14:9300", "192.168.100.15:9300", "192.168.100.19:9300"]
cluster.initial_master_nodes: ["node-1"]
192.168.100.19  config/elasticsearch.yml

cluster.name: my-monitor
node.name: node-3
network.host: 192.168.100.19
http.port: 9200
discovery.seed_hosts: ["192.168.100.14:9300", "192.168.100.15:9300", "192.168.100.19:9300"]
cluster.initial_master_nodes: ["node-1"]

At the same time, it is recommended to modify the config/jvm.options of the three nodes

-Xms2g
-Xmx2g

Start three nodes in sequence

pkill -F pid
./bin/elasticsearch -d -p pid

Next, modify the es address in config/application.yml under skywalking

storage:
  selector: ${SW_STORAGE:elasticsearch7}
  elasticsearch7:
    nameSpace: ${SW_NAMESPACE:""}
    clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:192.168.100.14:9200,192.168.100.15:9200,192.168.100.19:9200}

Install Agent

https://github.com/apache/skywalking/blob/v8.2.0/docs/en/setup/service-agent/java-agent/README.md
Copy the agent directory to the machine where each service is located

scp -r ./agent chengjs@192.168.100.12:~/

Here, I copy it to each service directory

Plugins are various plug-ins used by probes. SkyWalking plug-ins are plug-and-play. You can put the plug-ins in optional-plugins into plugins, modify the agent/config/agent.config configuration file, or pass command line parameters. Specify.

Mainly configure the service name and back-end service address

agent.service_name=${SW_AGENT_NAME:user-center}
collector.backend_service=${SW_AGENT_COLLECTOR_BACKEND_SERVICES:192.168.100.17:11800}

Of course, it can also be set through environment variables or system properties, for example:

export SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800

Finally, use the command line parameter -javaagent to specify the probe when the service starts

java -javaagent:/path/to/skywalking-agent/skywalking-agent.jar -jar yourApp.jar

E.g:

java -javaagent:./agent/skywalking-agent.jar -Dspring.profiles.active=dev -Xms512m -Xmx1024m -jar demo-0.0.1-SNAPSHOT.jar

start service

Modify the webapp/webapp.yml file, change the port number and back-end service address

server:
  port: 9000

collector:
  path: /graphql
  ribbon:
    ReadTimeout: 10000
    # Point to all backend's restHost:restPort, split by ,
    listOfServers: 127.0.0.1:12800

Start service

bin/startup.sh

Or start separately

bin/oapService.sh
bin/webappService.sh

Check the log file in the logs directory to see if the startup is successful. The browser visits http://127.0.0.1:9000 .

alarm

Edit alarm-settings.yml to set alarm rules and notifications

https://github.com/apache/skywalking/blob/v8.2.0/docs/en/setup/backend/backend-alarm.md

Focus on the alarm notification

In order to use Dingding robot notification, next, create a new project

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.4.0</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.wt.monitor</groupId>
    <artifactId>skywalking-alarm</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <name>skywalking-alarm</name>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>com.aliyun</groupId>
            <artifactId>alibaba-dingtalk-service-sdk</artifactId>
            <version>1.0.1</version>
        </dependency>

        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>1.15</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.75</version>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Optional dependencies (not recommended)

<dependency
    <groupId>org.apache.skywalking</groupId>
    <artifactId>server-core</artifactId>
    <version>8.2.0</version>
</dependency>

Define the alert message entity class

package com.wt.monitor.skywalking.alarm.domain;

import lombok.Data;

import java.io.Serializable;

/**
 * @author ChengJianSheng
 * @date 2020/12/1
 */
@Data
public class AlarmMessageDTO implements Serializable {

    private int scopeId;

    private String scope;

    /**
     * Target scope entity name
     */
    private String name;

    private String id0;

    private String id1;

    private String ruleName;

    /**
     * Alarm text message
     */
    private String alarmMessage;

    /**
     * Alarm time measured in milliseconds
     */
    private long startTime;

}

Send Dingding robot message

package com.wt.monitor.skywalking.alarm.service;

import com.dingtalk.api.DefaultDingTalkClient;
import com.dingtalk.api.DingTalkClient;
import com.dingtalk.api.request.OapiRobotSendRequest;
import com.taobao.api.ApiException;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.codec.binary.Base64;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;

/**
 * https://ding-doc.dingtalk.com/doc#/serverapi2/qf2nxq
 * @author ChengJianSheng
 * @data 2020/12/1
 */
@Slf4j
@Service
public class DingTalkAlarmService {

    @Value("${dingtalk.webhook}")
    private String webhook;
    @Value("${dingtalk.secret}")
    private String secret;

    public void sendMessage(String content) {
        try {
            Long timestamp = System.currentTimeMillis();
            String stringToSign = timestamp + "\n" + secret;
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(secret.getBytes("UTF-8"), "HmacSHA256"));
            byte[] signData = mac.doFinal(stringToSign.getBytes("UTF-8"));
            String sign = URLEncoder.encode(new String(Base64.encodeBase64(signData)),"UTF-8");

            String serverUrl = webhook + "&timestamp=" + timestamp + "&sign=" + sign;
            DingTalkClient client = new DefaultDingTalkClient(serverUrl);
            OapiRobotSendRequest request = new OapiRobotSendRequest();
            request.setMsgtype("text");
            OapiRobotSendRequest.Text text = new OapiRobotSendRequest.Text();
            text.setContent(content);
            request.setText(text);

            client.execute(request);
        } catch (ApiException e) {
            e.printStackTrace();
            log.error(e.getMessage(), e);
        } catch (NoSuchAlgorithmException e) {
            e.printStackTrace();
            log.error(e.getMessage(), e);
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
            log.error(e.getMessage(), e);
        } catch (InvalidKeyException e) {
            e.printStackTrace();
            log.error(e.getMessage(), e);
        }
    }
}

AlarmController.java

package com.wt.monitor.skywalking.alarm.controller;

import com.alibaba.fastjson.JSON;
import com.wt.monitor.skywalking.alarm.domain.AlarmMessageDTO;
import com.wt.monitor.skywalking.alarm.service.DingTalkAlarmService;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import java.text.MessageFormat;
import java.util.List;

/**
 * @author ChengJianSheng
 * @date 2020/12/1
 */
@Slf4j
@RestController
@RequestMapping("/skywalking")
public class AlarmController {

    @Autowired
    private DingTalkAlarmService dingTalkAlarmService;

    @PostMapping("/alarm")
    public void alarm(@RequestBody List<AlarmMessageDTO> alarmMessageDTOList) {
       log.info("收到告警信息: {}", JSON.toJSONString(alarmMessageDTOList));
       if (null != alarmMessageDTOList) {
           alarmMessageDTOList.forEach(e->dingTalkAlarmService.sendMessage(MessageFormat.format("-----来自SkyWalking的告警-----\n【名称】: {0}\n【消息】: {1}\n", e.getName(), e.getAlarmMessage())));
       }
    }
}

Author: Big Brother Waste
Original: https://www.cnblogs.com/cjsblog/p/14075486.html