Skywalking plugin development

The source of this article is shenyifengtk.github.io , please explain

concept

Span

Span is an important and commonly used concept in distributed tracing systems. Learn more about spans from Google Dapper Paper and OpenTracing .

SkyWalking supports OpenTracing and OpenTracing-Java API since 2017, our Span concept is similar to the paper and OpenTracing. We also extend Span.

There are three types of Span

1.1 EntrySpan

EntrySpan represents service provider and is also the endpoint on the server side. As an APM system, our target is the application server. So almost all services and MQ-consumers are EntrySpans. It can be understood that the first span processed by a process is EntrySpan, which means that the entire span enters the service span.

1.2 LocalSpan

LocalSpan represents a normal Java method that has nothing to do with remote services, nor is it an MQ producer/consumer, nor a service (eg HTTP service) provider/consumer. All local method calls are localSpan, including asynchronous thread calls, thread pool submission tasks are.

1.3 ExitSpan

ExitSpan represents a service client or MQ producer, named LeafSpan in the early days of SkyWalking. For example, accessing DB through JDBC, reading Redis/Memcached is classified as ExitSpan.

ContextCarrier

In order to achieve distributed tracing, it is necessary to bind traces across processes, and the context should be propagated with it throughout the process. This is the responsibility of the ContextCarrier.

Here are the steps on how to use ContextCarrier in A -> B distributed calls.

1. On the client side, create a new empty ContextCarrier .
2. Create an ExitSpan by ContextManager#createExitSpan or use ContextManager#inject to initialize ContextCarrier .
3. Put ContextCarrier all the information in the request header (such as HTTP HEAD), attachments (such as Dubbo RPC framework), or messages (such as Kafka). For details, see the official cross-process transmission protocol sw8
4. Through the service call, pass ContextCarrier to the server.
5. On the server side, get ContextCarrier all content in the header, attachment or message of the corresponding component.
6. Create an EntrySpan by ContestManager#createEntrySpan or use ContextManager#extract to bind the server and the client.

Let's demonstrate through Apache HttpComponent client plugin and Tomcat 7 server plugin, the steps are as follows:

1. Client Apache HttpComponent client plugin

 span = ContextManager.createExitSpan("/span/operation/name", contextCarrier, "ip:port");
CarrierItem next = contextCarrier.items();
while (next.hasNext()) {
    next = next.next();
    httpRequest.setHeader(next.getHeadKey(), next.getHeadValue());
}

2. Server Tomcat 7 server plugin

 ContextCarrier contextCarrier = new ContextCarrier();
CarrierItem next = contextCarrier.items();
while (next.hasNext()) {
    next = next.next();
    next.setHeadValue(request.getHeader(next.getHeadKey()));
}

span = ContextManager.createEntrySpan("/span/operation/name", contextCarrier);

ContextSnapshot

In addition to cross-process, cross-thread also needs to be supported. For example, asynchronous threads (message queues in memory) and batch processing are very common in Java. Cross-process and cross-thread are very similar, because they both need to propagate the context. The only difference is, Cross-thread serialization is not required.

Here are the three steps for spreading across threads:

1. Use the ContextManager#capture method to get the ContextSnapshot object.
2. Let the child thread access the ContextSnapshot in any way, either through method parameters or carried by existing parameters
3. Use ContextManager#continued in the child thread.

Cross-process Span transmission principle

 public class CarrierItem implements Iterator<CarrierItem> {
    private String headKey;
    private String headValue;
    private CarrierItem next;

    public CarrierItem(String headKey, String headValue) {
        this(headKey, headValue, null);
    }

    public CarrierItem(String headKey, String headValue, CarrierItem next) {
        this.headKey = headKey;
        this.headValue = headValue;
        this.next = next;
    }

    public String getHeadKey() {
        return headKey;
    }

    public String getHeadValue() {
        return headValue;
    }

    public void setHeadValue(String headValue) {
        this.headValue = headValue;
    }

    @Override
    public boolean hasNext() {
        return next != null;
    }

    @Override
    public CarrierItem next() {
        return next;
    }

    @Override
    public void remove() {

    }
}

CarrierItem is a data interface similar to Map key value, connecting K/V through a one-way connection.
See how the ContextCarrier.items() method creates a CarrierItem

 public CarrierItem items() {
       //内置一个 sw8-x key
        SW8ExtensionCarrierItem sw8ExtensionCarrierItem = new SW8ExtensionCarrierItem(extensionContext, null); 
       //内置  sw8-correlation key
        SW8CorrelationCarrierItem sw8CorrelationCarrierItem = new SW8CorrelationCarrierItem(
            correlationContext, sw8ExtensionCarrierItem);
       //内置 sw8 key 
        SW8CarrierItem sw8CarrierItem = new SW8CarrierItem(this, sw8CorrelationCarrierItem);
        return new CarrierItemHead(sw8CarrierItem);
    }

Create a link CarrierItemHead->SW8CarrierItem->SW8CorrelationCarrierItem->SW8ExtensionCarrierItem
Look at the above tomcat7 traverse the CarrierItem, call the key to get the value from the http header and set it to the built-in value of the object, so that the header value of the previous process can be set to the next process.

 ContextCarrier deserialize(String text, HeaderVersion version) {
        if (text == null) {
            return this;
        }
        if (HeaderVersion.v3.equals(version)) {
            String[] parts = text.split("-", 8);
            if (parts.length == 8) {
                try {
                    // parts[0] is sample flag, always trace if header exists.
                    this.traceId = Base64.decode2UTFString(parts[1]);
                    this.traceSegmentId = Base64.decode2UTFString(parts[2]);
                    this.spanId = Integer.parseInt(parts[3]);
                    this.parentService = Base64.decode2UTFString(parts[4]);
                    this.parentServiceInstance = Base64.decode2UTFString(parts[5]);
                    this.parentEndpoint = Base64.decode2UTFString(parts[6]);
                    this.addressUsedAtClient = Base64.decode2UTFString(parts[7]);
                } catch (IllegalArgumentException ignored) {

                }
            }
        }
        return this;
    }

In this way, the newly created ContextCarrier can inherit all the properties from the previous caller, and the newly created span can be associated with the previous span.

develop plugin

Knowledge point

The basic method of tracing is to intercept Java methods, using byte-buddy and AOP concepts. SkyWalking wraps byte-buddy and traces the propagation of context, so you only need to define interception points (in other words, Aspects of Spring).

ClassInstanceMethodsEnhancePluginDefine defines the constructor interception point and instance method instance method interception point, there are three main methods that need to be rewritten

 /**
     * 需要被拦截Class
     * @return
     */
    @Override
    protected ClassMatch enhanceClass() {
        return null;
    }

    /**
     * 构造器切点
     * @return
     */
    @Override
    public ConstructorInterceptPoint[] getConstructorsInterceptPoints() {
        return new ConstructorInterceptPoint[0];
    }

    /**
     * 方法切点
     * @return InstanceMethodsInterceptPoint 里面会声明拦截按个方法
     */
    @Override
    public InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints() {
        return new InstanceMethodsInterceptPoint[0];
    }

ClassMatch The following four methods indicate how to match the target class:

• NameMatch.byName , through the fully qualified class name (Fully Qualified Class Name, that is, the package name + . + class name).
• ClassAnnotationMatch.byClassAnnotationMatch , according to whether certain annotations exist in the target class.
• MethodAnnotationMatchbyMethodAnnotationMatch , according to whether there are some annotations for the method of the target class.
• HierarchyMatch.byHierarchyMatch , according to the parent class or interface of the target class

ClassStaticMethodsEnhancePluginDefine defines the class method static method interception point.

 public abstract class ClassStaticMethodsEnhancePluginDefine extends ClassEnhancePluginDefine {

    /**
     * 构造器切点
     * @return null, means enhance no constructors.
     */
    @Override
    public ConstructorInterceptPoint[] getConstructorsInterceptPoints() {
        return null;
    }

    /**
     * 方法切点
     * @return null, means enhance no instance methods.
     */
    @Override
    public InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints() {
        return null;
    }
}

InstanceMethodsInterceptPoint What methods are there for common method interface pointcuts?

 public interface InstanceMethodsInterceptPoint {
    /**
     * class instance methods matcher.
     *  可以理解成功对Class 那些方法进行增强
     *  ElementMatcher 是bytebuddy 类库一个方法匹配器，里面封装了各种方法匹配
     * @return methods matcher
     */
    ElementMatcher<MethodDescription> getMethodsMatcher();

    /**
     * @return represents a class name, the class instance must instanceof InstanceMethodsAroundInterceptor.
     *  返回一个拦截器全类名，所有拦截器必须实现    InstanceMethodsAroundInterceptor 接口
     */
    String getMethodsInterceptor();

    /**
     *  是否要覆盖原方法入参
     * @return
     */
    boolean isOverrideArgs();
}

Look at the methods of the interceptor

 /**
 * A interceptor, which intercept method's invocation. The target methods will be defined in {@link
 * ClassEnhancePluginDefine}'s subclass, most likely in {@link ClassInstanceMethodsEnhancePluginDefine}
 */
public interface InstanceMethodsAroundInterceptor {
    /**
     * called before target method invocation.
     * 前置通知
     * @param result change this result, if you want to truncate the method.
     */
    void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
        MethodInterceptResult result) throws Throwable;

    /**
     * called after target method invocation. Even method's invocation triggers an exception.
     * 后置通知
     * @param ret the method's original return value. May be null if the method triggers an exception.
     * @return the method's actual return value.
     */
    Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
        Object ret) throws Throwable;

    /**
     * called when occur exception.
     * 异常通知
     * @param t the exception occur.
     */
    void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments,
        Class<?>[] argumentsTypes, Throwable t);
}

Develop Skywalking actual combat

Project maven environment configuration

 <?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>tk.shenyifeng</groupId>
    <artifactId>skywalking-plugin</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <skywalking.version>8.10.0</skywalking.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.apache.skywalking</groupId>
            <artifactId>apm-agent-core</artifactId>
            <version>${skywalking.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.skywalking</groupId>
            <artifactId>java-agent-util</artifactId>
            <version>${skywalking.version}</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>


    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>8</source>
                    <target>8</target>
                </configuration>
            </plugin>

            <plugin>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <shadedArtifactAttached>false</shadedArtifactAttached>
                            <createDependencyReducedPom>true</createDependencyReducedPom>
                            <createSourcesJar>true</createSourcesJar>
                            <shadeSourcesContent>true</shadeSourcesContent>
                            <relocations>
                                <relocation>
                                    <pattern>net.bytebuddy</pattern>
                                    <shadedPattern>org.apache.skywalking.apm.dependencies.net.bytebuddy</shadedPattern>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

In order to be more representative, use the ES plugin officially developed by Skywalking as an example. In order to be compatible with different versions of the framework, Skywalking officially uses witnessClasses. If this Class exists in the current framework Jar, the task will be a certain version. Similarly, the witnessMethods will have a certain Method in the Class.

 public class AdapterActionFutureInstrumentation extends ClassEnhancePluginDefine {

    @Override
    public ConstructorInterceptPoint[] getConstructorsInterceptPoints() {
        return new ConstructorInterceptPoint[0];
    }

    @Override
    public InstanceMethodsInterceptPoint[] getInstanceMethodsInterceptPoints() {
        return new InstanceMethodsInterceptPoint[] {
            new InstanceMethodsInterceptPoint() {
                @Override
                public ElementMatcher<MethodDescription> getMethodsMatcher() {
                    return named("actionGet"); //拦截方法
                }

                @Override
                public String getMethodsInterceptor() {  //拦截器全类名
                    return "org.apache.skywalking.apm.plugin.elasticsearch.v7.interceptor.AdapterActionFutureActionGetMethodsInterceptor";
                }

                @Override
                public boolean isOverrideArgs() {
                    return false;
                }
            }
        };
    }

    @Override
    public StaticMethodsInterceptPoint[] getStaticMethodsInterceptPoints() {
        return new StaticMethodsInterceptPoint[0];
    }

    @Override
    protected ClassMatch enhanceClass() { //增强Class
        return byName("org.elasticsearch.action.support.AdapterActionFuture");
    }

    @Override
    protected String[] witnessClasses() {//ES7 存在Class
        return new String[] {"org.elasticsearch.transport.TaskTransportChannel"};
    }

    @Override
    protected List<WitnessMethod> witnessMethods() { //ES7 SearchHits 存在方法
        return Collections.singletonList(new WitnessMethod(
            "org.elasticsearch.search.SearchHits",
          named("getTotalHits").and(takesArguments(0)).and(returns(named("org.apache.lucene.search.TotalHits")))
        ));
    }
}

Creates an interceptor with a given class name that implements the InstanceMethodsAroundInterceptor interface. Create an EntrySpan

 public class TomcatInvokeInterceptor implements InstanceMethodsAroundInterceptor {

    private static boolean IS_SERVLET_GET_STATUS_METHOD_EXIST;
    private static final String SERVLET_RESPONSE_CLASS = "javax.servlet.http.HttpServletResponse";
    private static final String GET_STATUS_METHOD = "getStatus";

    static {
        IS_SERVLET_GET_STATUS_METHOD_EXIST = MethodUtil.isMethodExist(
            TomcatInvokeInterceptor.class.getClassLoader(), SERVLET_RESPONSE_CLASS, GET_STATUS_METHOD);
    }

    /**
     * * The {@link TraceSegment#ref} of current trace segment will reference to the trace segment id of the previous
     * level if the serialized context is not null.
     *
     * @param result change this result, if you want to truncate the method.
     */
    @Override
    public void beforeMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
                             MethodInterceptResult result) throws Throwable {
        Request request = (Request) allArguments[0];
        ContextCarrier contextCarrier = new ContextCarrier();

        CarrierItem next = contextCarrier.items();
       //如果 HTTP 请求头中有符合sw8 传输协议的请求头则 取出来设置到上下文ContextCarrier
        while (next.hasNext()) {
            next = next.next();
            next.setHeadValue(request.getHeader(next.getHeadKey()));
        }
        String operationName =  String.join(":", request.getMethod(), request.getRequestURI());
        AbstractSpan span = ContextManager.createEntrySpan(operationName, contextCarrier);//关联起来
        Tags.URL.set(span, request.getRequestURL().toString()); //添加 span 参数
        Tags.HTTP.METHOD.set(span, request.getMethod());
        span.setComponent(ComponentsDefine.TOMCAT);
        SpanLayer.asHttp(span);

        if (TomcatPluginConfig.Plugin.Tomcat.COLLECT_HTTP_PARAMS) {
            collectHttpParam(request, span);
        }
    }

    @Override
    public Object afterMethod(EnhancedInstance objInst, Method method, Object[] allArguments, Class<?>[] argumentsTypes,
                              Object ret) throws Throwable {
        Request request = (Request) allArguments[0];
        HttpServletResponse response = (HttpServletResponse) allArguments[1];

        AbstractSpan span = ContextManager.activeSpan();
        if (IS_SERVLET_GET_STATUS_METHOD_EXIST && response.getStatus() >= 400) {
            span.errorOccurred();
            Tags.HTTP_RESPONSE_STATUS_CODE.set(span, response.getStatus());
        }
        // Active HTTP parameter collection automatically in the profiling context.
        if (!TomcatPluginConfig.Plugin.Tomcat.COLLECT_HTTP_PARAMS && span.isProfiling()) {
            collectHttpParam(request, span);
        }
        ContextManager.getRuntimeContext().remove(Constants.FORWARD_REQUEST_FLAG);
        ContextManager.stopSpan();
        return ret;
    }

    @Override
    public void handleMethodException(EnhancedInstance objInst, Method method, Object[] allArguments,
                                      Class<?>[] argumentsTypes, Throwable t) {
        AbstractSpan span = ContextManager.activeSpan();
        span.log(t);
    }

    private void collectHttpParam(Request request, AbstractSpan span) {
        final Map<String, String[]> parameterMap = new HashMap<>();
        final org.apache.coyote.Request coyoteRequest = request.getCoyoteRequest();
        final Parameters parameters = coyoteRequest.getParameters();
        for (final Enumeration<String> names = parameters.getParameterNames(); names.hasMoreElements(); ) {
            final String name = names.nextElement();
            parameterMap.put(name, parameters.getParameterValues(name));
        }

        if (!parameterMap.isEmpty()) {
            String tagValue = CollectionUtil.toString(parameterMap);
            tagValue = TomcatPluginConfig.Plugin.Http.HTTP_PARAMS_LENGTH_THRESHOLD > 0 ?
                StringUtil.cut(tagValue, TomcatPluginConfig.Plugin.Http.HTTP_PARAMS_LENGTH_THRESHOLD) :
                tagValue;
            Tags.HTTP.PARAMS.set(span, tagValue);
        }
    }
}

After developing the interceptor, be sure to add the skywalking-plugin.def file to the classpath, and add the full class name after development to the configuration.

xxxName = tk.shenyifeng.skywalking.plugin.RepladInstrumentation

If this file is not present in the jar, the plugin will not be loaded by Skywalking.
Finally, put the packaged jar in the plugin or activations directory of Skywalking.

xml configuration plugin

 <?xml version="1.0" encoding="UTF-8"?>
<enhanced>
    <class class_name="test.apache.skywalking.apm.testcase.customize.service.TestService1">
        <method method="staticMethod()" operation_name="/is_static_method" static="true"></method>
        <method method="staticMethod(java.lang.String,int.class,java.util.Map,java.util.List,[Ljava.lang.Object;)"
                operation_name="/is_static_method_args" static="true">
            <operation_name_suffix>arg[0]</operation_name_suffix>
            <operation_name_suffix>arg[1]</operation_name_suffix>
            <operation_name_suffix>arg[3].[0]</operation_name_suffix>
            <tag key="tag_1">arg[2].['k1']</tag>
            <tag key="tag_2">arg[4].[1]</tag>
            <log key="log_1">arg[4].[2]</log>
        </method>
        <method method="method()" static="false"></method>
        <method method="method(java.lang.String,int.class)" operation_name="/method_2" static="false">
            <operation_name_suffix>arg[0]</operation_name_suffix>
            <tag key="tag_1">arg[0]</tag>
            <log key="log_1">arg[1]</log>
        </method>
        <method
            method="method(test.apache.skywalking.apm.testcase.customize.model.Model0,java.lang.String,int.class)"
            operation_name="/method_3" static="false">
            <operation_name_suffix>arg[0].id</operation_name_suffix>
            <operation_name_suffix>arg[0].model1.name</operation_name_suffix>
            <operation_name_suffix>arg[0].model1.getId()</operation_name_suffix>
            <tag key="tag_os">arg[0].os.[1]</tag>
            <log key="log_map">arg[0].getM().['k1']</log>
        </method>
        <method method="retString(java.lang.String)" operation_name="/retString" static="false">
            <tag key="tag_ret">returnedObj</tag>
            <log key="log_map">returnedObj</log>
        </method>
        <method method="retModel0(test.apache.skywalking.apm.testcase.customize.model.Model0)"
          operation_name="/retModel0" static="false">
            <tag key="tag_ret">returnedObj.model1.id</tag>
            <log key="log_map">returnedObj.model1.getId()</log>
        </method>
    </class>
    
</enhanced>

Through xml configuration, you can save the steps of writing Java code and packaging jar.
xml rules

configure	illustrate
class_name	Need to be enhanced Class
method	Method needs to be enhanced to support parameter definition
operation_name	Action name
operation_name_suffix	Operation suffix, used to generate dynamic operation_name
tag	will add a tag to the local span. The value of the key needs to be represented on the XML node
log	will add a log to the local span. The value of the key needs to be represented on the XML node
arg[n]	Represents the input parameter value. For example, args[0] represents the first parameter
.[n]	When the object being parsed is an Array or List, you can use this expression to get the object at the corresponding index
.['key']	When the object being parsed is a Map, you can use this expression to get the key of the map

Add configuration in the configuration file agent.config:

plugin.customize.enhance_file=absolute path to customize_enhance.xml

---

Citations
https://www.itmuch.com/skywalking/apm-customize-enhance-plugin/
https://skyapm.github.io/document-cn-translation-of-skywalking/en/6.1.0/guides/Java-Plugin-Development-Guide.html