Use Java diagnostic tool-Arthas in docker container

I. Overview

Deploying projects using docker containers has become a skill that developers must master. After deploying projects using docker containers, how to diagnose Java applications in the container in real time. This article mainly introduces how to use Java diagnostic tools in docker containers-Arthas . There is not much difference between using Arthas in a container and using it on a server. Normally, only one Java process of our application service will run in a container, so using Arthas in a container will only see one Java process. For a detailed description of Arthas, you can check the official document below. Here, I will only introduce the Arthas commands that I have used in the docker container through cases.

The content of this article is as follows:

• Modify the log level in real time
• Real-time view method call input and output parameters
• real-time online hot update code

Arthas Chinese document: http://arthas.gitee.io/

Second, install Arthas in the docker container

1. Build a docker container

Here is a spring boot demo for practice, which only includes a controller, the content is as follows:

@Slf4j
@RestController
public class TestController {

    @GetMapping("hello/{content}")
    public String hello(@PathVariable(value = "content") String content) {
        log.debug("----------log debug----------");
        log.info("----------log info----------");
        log.warn("----------log warn----------");
        log.error("----------log error----------");
        return "返回结果：" + content;
    }
}

The content of the Dockerfile to build the image is as follows:

FROM openjdk:8u232-jdk
WORKDIR /app
LABEL maintainer="peterwd" app="devops-demo"
COPY target/devops-demo.jar devops-demo.jar
EXPOSE 8080
CMD java -jar devops-demo.jar

Use the following command to build the image:

docker build -t devops-demo .

Use the following command to start the container:

docker run --name devop-demo -d -p 8080:8080 devops-demo

After building the image, use the following command to enter the docker container:

docker exec -it devops-demo bash

2. Install Arthas

After entering the docker container, use the following command to install Arthas:

wget https://arthas.aliyun.com/arthas-boot.jar

Use the following command to start Arthas:

java -jar arthas-boot.jar

In the process of starting Arthas, the corresponding Java process will be selected. There is usually only one Java process in the docker container, so just go to 1. If there are multiple Java processes, enter the previous number.
As shown below:

Three, Arthas command introduction

Please note that these commands are implemented through bytecode enhancement technology. Some aspects will be inserted into the methods of the specified class to achieve data statistics and observation. Therefore, when using online and pre-release, please try to be clear about the class that needs to be observed. , Methods and conditions. After the diagnosis is finished, execute stop or execute the reset command for the enhanced class.

1. Basic commands

help——View command help information

cat-print the content of the file, similar to the cat command in linux

echo-print parameters, similar to the echo command in linux

grep-matching search, similar to the grep command in linux

base64-base64 encoding conversion, similar to the base64 command in linux

tee-copy standard input to standard output and the specified file, similar to the tee command in linux

pwd-returns the current working directory, similar to the linux command

cls-Clear the current screen area

session-view information about the current session

reset——reset the enhanced classes, all the classes enhanced by Arthas will be restored, and all the enhanced classes will be reset when the Arthas server is closed

version-output the version number of Arthas loaded by the current target Java process

history-print command history

quit——Quit the current Arthas client, other Arthas clients will not be affected

stop——Close the Arthas server, and all Arthas clients will exit

keymap-Arthas shortcut key list and custom shortcut keys

2. Jvm related commands

dashboard-the real-time data panel of the current system

thread-View the thread stack information of the current JVM

jvm-view current JVM information

sysprop-view and modify the system properties of the JVM

sysenv-view the environment variables of the JVM

vmoption-view and modify diagnostic-related options in the JVM

perfcounter-View the Perf Counter information of the current JVM

logger-view and modify logger

getstatic-view the static properties of the class

ognl-execute ognl expression

mbean-view information about Mbean

heapdump-dump java heap, heap dump function similar to jmap command

vmtool-query objects from jvm, execute forceGc

3. Class/classloader related commands

sc-View the loaded class information of the JVM

sm-view the method information of the loaded class

jad-Decompile the source code of the specified loaded class

mc-memory compiler, memory compiles .java files into .class files

retransform-load external .class files and retransform to JVM

redefine-load external .class files and redefine to JVM

dump——dump the byte code of the loaded class to a specific directory

classloader-view classloader inheritance tree, urls, class loading information, use classloader to getResource

4. Monitor/watch/trace related commands

monitor-method execution monitoring

watch-method to perform data observation

trace-method internal call path, and output the time-consuming on each node on the method path

stack-output the call path of the current method being called

tt——The space-time tunnel of method execution data, record the input parameters and return information of each call of the specified method, and can observe these calls at different times

3. Use Arthas' logger modify the log level of the class in real time

A brief introduction to Arthas commands is introduced here. Here we mainly introduce the use of Arthas' logger modify the log level of the class in real time. The demo used here defines four log levels, namely debug、info、warn、error . The display of logs is controlled by dynamically modifying different log levels. .

log4j defines 8 levels of log (excluding OF F and ALL, it can be said to be divided into 6 levels), the priority from high to low is as follows:
OFF、FATAL、ERROR、WARN、INFO、DEBUG、TRACE、 ALL。

If the log level is set at a certain level, then the log with a higher priority than this level can be printed . For example, if the priority is set to WARN, the logs of the 4 levels of OFF, FATAL, ERROR, and WARN can be output normally, while the logs of the INFO, DEBUG, TRACE, and ALL levels will be ignored. Log4j recommends to use only four levels, the priority from high to low is ERROR、WARN、INFO、DEBUG .

1. Use the sc command to view the class information loaded by the JVM

command: sc -d [find the full path of the class or * class name]

sc command supports fuzzy search of class information by class name. -d displays detailed information. The full path name of the class and classLoaderHash are as follows:

[arthas@7]$ sc -d *TestController 
 class-info        devops.demo.controller.TestController
 code-source       file:/app/devops-demo.jar!/BOOT-INF/classes!/
 name              devops.demo.controller.TestController
 isInterface       false
 isAnnotation      false
 isEnum            false
 isAnonymousClass  false
 isArray           false
 isLocalClass      false
 isMemberClass     false
 isPrimitive       false
 isSynthetic       false
 simple-name       TestController
 modifier          public
 annotation        org.springframework.web.bind.annotation.RestController
 interfaces
 super-class       +-java.lang.Object
 class-loader      +-org.springframework.boot.loader.LaunchedURLClassLoader@7daf6ecc
                     +-sun.misc.Launcher$AppClassLoader@70dea4e
                       +-sun.misc.Launcher$ExtClassLoader@1a04f701
 classLoaderHash   7daf6ecc    

2. Use the logger command to view the log level of the specified class

command: logger --name [find the full path of the class]

logger used to view and modify logger information, --name specifies the full path class name, as shown below:

[arthas@7]$ logger --name devops.demo.controller.TestController 
 name                               devops.demo.controller.TestController                                                                                                                                        
 class                              ch.qos.logback.classic.Logger                                                                                                                                                
 classLoader                        org.springframework.boot.loader.LaunchedURLClassLoader@7daf6ecc                                                                                                              
 classLoaderHash                    7daf6ecc                                                                                                                                                                     
 level                              null                                                                                                                                                                         
 effectiveLevel                     INFO                                                                                                                                                                         
 additivity                         true                                                                                                                                                                         
 codeSource                         jar:file:/app/devops-demo.jar!/BOOT-INF/lib/logback-classic-1.2.3.jar!/    

3. Use the logger command to modify the log level of the specified class

command: logger -c [classLoaderHash value] --name [find the full path of the class] --level [log level to be updated]

-c specifies the value of classLoaderHash --level specifies the log level to be updated, as shown below:

[arthas@7]$ logger -c 7daf6ecc --name devops.demo.controller.TestController --level debug
Update logger level success.

4. Verify the result of modifying the log level

By default, the log level of the class is info . There is no debug information in the demo output log here, as shown in the following figure:

Use the following command to modify the log level of logger to debug:

[arthas@7]$ logger -c 7daf6ecc --name devops.demo.controller.TestController --level debug
Update logger level success.

Visit again, the output log has debug information, as shown in the figure below:

Use the following command to modify the log level of logger to error:

[arthas@7]$ logger -c 7daf6ecc --name devops.demo.controller.TestController --level error
Update logger level success.

Visit again, the output log only has error information, as shown in the figure below:

Fourth, use Arthas' watch view method input and output parameters

command: watch full path class name method name [expression]

watch for using the 060b1e67ed3237 command are as follows:
watch used to view the input and output parameters, return value and exception information of the specified method call. watch can use are as follows:

target : the object
clazz : the object's class
method : the constructor or method
params : the parameters array of method
params[0..n] : the element of parameters array
returnObj : the returned object of method
throwExp : the throw exception of method
isReturn : the method ended by return
isThrow : the method ended by throwing exception
#cost : the execution time in ms of method invocation

watch the detailed description of the 060b1e67ed3267 command, you can use watch --help view, here only the use of the example method is introduced.

Use the following command to view the call parameters of the method:

[arthas@7]$ watch devops.demo.controller.TestController hello params
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 26 ms, listenerId: 3
method=devops.demo.controller.TestController.hello location=AtExit
ts=2021-05-26 11:36:58; [cost=0.627099ms] result=@Object[][
    @String[测试方法调用参数],
]

Use the following command to view the return parameters of the method:

[arthas@7]$ watch devops.demo.controller.TestController hello returnObj 
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 24 ms, listenerId: 4
method=devops.demo.controller.TestController.hello location=AtExit
ts=2021-05-26 11:39:18; [cost=0.525488ms] result=@String[返回结果：测试方法返回参数]

Five, use Arthas to achieve online code hot update

You can use the sc jad mc redefine provided by Arthas to implement online code hot update. This function is very powerful, but it is also very dangerous. You need to control the access to the container when you use the container, and you must also control the server's use permission when you use the server. Take the provided demo as an example to explain in detail how to use these commands to implement hot code updates.

1. Use the sc command to find the class information loaded by the JVM

command: sc -d [find the full path of the class or * class name]

sc command supports fuzzy search of class information by class name. -d displays detailed information. The full path name of the class and classLoaderHash are as follows:

[arthas@7]$ sc -d *TestController 
 class-info        devops.demo.controller.TestController
 code-source       file:/app/devops-demo.jar!/BOOT-INF/classes!/
 name              devops.demo.controller.TestController
 isInterface       false
 isAnnotation      false
 isEnum            false
 isAnonymousClass  false
 isArray           false
 isLocalClass      false
 isMemberClass     false
 isPrimitive       false
 isSynthetic       false
 simple-name       TestController
 modifier          public
 annotation        org.springframework.web.bind.annotation.RestController
 interfaces
 super-class       +-java.lang.Object
 class-loader      +-org.springframework.boot.loader.LaunchedURLClassLoader@7daf6ecc
                     +-sun.misc.Launcher$AppClassLoader@70dea4e
                       +-sun.misc.Launcher$ExtClassLoader@1a04f701
 classLoaderHash   7daf6ecc    

2. Use the jad command to decompile the source code of the loaded class

command: jad --source-only full path of the class> class name.java

jad command decompiles the source code of the loaded class, --source-only specifies that only the source code is output, > class name.java saves the output result to the class name.java file in the current directory

jad --source-only devops.demo.controller.TestController > TestController.java

View the contents of the decompilation as follows:

[arthas@7]$ cat TestController.java 
       /*
        * Decompiled with CFR.
        * 
        * Could not load the following classes:
        *  org.slf4j.Logger
        *  org.slf4j.LoggerFactory
        *  org.springframework.web.bind.annotation.GetMapping
        *  org.springframework.web.bind.annotation.PathVariable
        *  org.springframework.web.bind.annotation.RestController
        */
       package devops.demo.controller;
       
       import org.slf4j.Logger;
       import org.slf4j.LoggerFactory;
       import org.springframework.web.bind.annotation.GetMapping;
       import org.springframework.web.bind.annotation.PathVariable;
       import org.springframework.web.bind.annotation.RestController;
       
       @RestController
       public class TestController {
           private static final Logger log = LoggerFactory.getLogger(TestController.class);
       
           @GetMapping(value={"hello/{content}"})
           public String hello(@PathVariable(value="content") String content) {
/*14*/         log.debug("----------log debug----------");
/*15*/         log.info("----------log info----------");
/*16*/         log.warn("----------log warn----------");
/*17*/         log.error("----------log error----------");
               return "返回结果：" + content;
           }
       }

There is no vim editor in the container, which is inconvenient to modify. You can copy the decompiled source code. After the modification is completed, use the docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH command to copy the modified content to the container.

docker cp TestController.java devop-demo:/app

Modify the source code decompiled above and modify it as follows:

[arthas@7]$ cat TestController.java 
       /*
        * Decompiled with CFR.
        * 
        * Could not load the following classes:
        *  org.slf4j.Logger
        *  org.slf4j.LoggerFactory
        *  org.springframework.web.bind.annotation.GetMapping
        *  org.springframework.web.bind.annotation.PathVariable
        *  org.springframework.web.bind.annotation.RestController
        */
       package devops.demo.controller;
       
       import org.slf4j.Logger;
       import org.slf4j.LoggerFactory;
       import org.springframework.web.bind.annotation.GetMapping;
       import org.springframework.web.bind.annotation.PathVariable;
       import org.springframework.web.bind.annotation.RestController;
       
       @RestController
       public class TestController {
           private static final Logger log = LoggerFactory.getLogger(TestController.class);
       
           @GetMapping(value={"hello/{content}"})
           public String hello(@PathVariable(value="content") String content) {
/*14*/         log.debug("----------log debug----------");
/*15*/         log.info("----------log info----------");
/*16*/         log.warn("----------log warn----------");
/*17*/         log.error("----------log error----------");
               return "返回结果：测试热更新代码 " + content;
           }
       }

3. Use mc memory to compile .java files into .class files

command mc -c class loader hash java source path -d /tmp

mc memory compiles the .java file as a .class file, -c class loader hash specifies the classLoaderHash found by the sc -d -d /tmp specifies that the directory of the compiled class file is /tmp , and if it is not specified, it will be output to the current directory.

[arthas@7]$ mc -c 7daf6ecc TestController.java  -d /tmp
Memory compiler output:
/tmp/devops/demo/controller/TestController.class
Affect(row-cnt:1) cost in 993 ms.

4. Use redefine load external .class files

command: redefine class file path

The class file path here fills in the path output from the decompilation above, as shown below:

[arthas@7]$ redefine /tmp/devops/demo/controller/TestController.class
redefine success, size: 1, classes:
devops.demo.controller.TestController

5. Verify the hot update results

root@devops-demo-7bdf65859c-mtjqm:/app# curl localhost:8080/hello/test
返回结果：测试热更新代码 test

Six, summary

This article briefly introduces the use of Arthas in the docker container. It mainly introduces the logger and watch commands and how to implement online code hot updates. Other commands will be used in the future to supplement them. For more information, please refer to the official documentation.