Exception handling that causes memory leaks

Memory Leak Due To Improper Exception Handling

https://dzone.com/articles/memory-leak-due-to-improper-exception-handling
Translation: Zhu Kunrong

In this article, we will discuss the memory problems we encountered in production and how to solve them. The app will become unresponsive after running for a few hours. But it's not clear what caused the app to become unresponsive.

technology stack

The application runs on an r5a.2xlarge EC2 instance in the AWS cloud. This application runs on an Apache Tomcat server using the Spring framework. It also uses AWS services like S3 and Elastic Beanstalk. The application uses a large heap size (-Xmx): 48GB.

position

We use the yCrash tool to locate this problem. We let the app run for 15 minutes of traffic. Then execute the yCrash script on this app. The yCrash script captures 360-degree data from the application stack, analyzes them, and shows the root cause of the problem. The data captured by the yCrash script includes: Garbage Collection logs, thread dump, heap dump, netstat, vmstat, iostat, top and ps.

The yCrash analysis material generates a memory leak report. The following is the heap dump analysis report generated by yCrash.

Figure 1: Large Object Report

You can see that yCrash points out that "org.apache.logging.log4j.LogManager" is the largest object in memory. Objects take up 98.2% of the total memory. Other objects take less than 2% of the memory. Here is the object tree for this largest object:

Figure 2: Object Reference Tree

Look at the red arrow in the object tree. This is the start code of the application. Part of the package names in Figure 2 are covered to prevent specific applications from being seen. You can see that this object package named "xxxxxxxx.superpower.Main$1.val$hprofParser" occupies 98.2% of the memory.
The application has a class called "xxxxxxxxxxxxxxx.Main.". Obviously the leak comes from this Main object. However, it is also not clear what "xxxxxxxxxxxxxxx.Main$1" is. "$1" indicates that this is the first anonymous inner class of the "xxxxxxxxxxxxxxx.Main" class. Anonymous inner class means that you can define an inner class without naming it in the parent class. But this is not a widely used Java programming practice. However, anonymous inner classes not only affect the readability of the program, but also lead to difficult positioning.

The following is the high-level summary source code of "xxxxxxxxxxxxxxx.Main". To reduce noise and improve readability, extraneous code in the class has been removed.

Figure 3: Source code causing memory leak

You can see that the ninth line is the anonymous inner class. This class extends the PrintingProgressMeter class. PrintingProgressMeter class inherits java.util.Thread. Any class that inherits java.util.Thread will become a thread.

On line 20, the PrintingProgressMeter thread is started by the pm.start() method; on line 21, the hprofParser.read() method is called; and on line 22, the thread is stopped by the pm.stopReporting() method. This code looks normal, right? What can trigger a memory leak in an application?

Problem: Exception Handling

In line 21 of hprofParser.read() there are certain scenarios where an exception may be thrown. If an exception is thrown, pm.stopReporting() on line 22 will not be called. If this line of code is not called, the thread will run forever and never exit. If the thread does not exit, the thread and object references (such as hprofParser) are not collected. It causes memory leaks.

solution

In most performance problems, locating the root cause of the problem is difficult. Fixing them is simple.
There is no exception here.

Figure 4: Source code to fix memory leak

We moved the pm.stopReporting() method to finally. In the Java language, code placed in a finally block will execute regardless of whether an exception is thrown. The content of the finally block can be found here https://docs.oracle.com/javase/tutorial/essential/exceptions/finally.html . In this way, even if the hprofParser.read() method throws an exception, the pm.stopReporting method will still be called to terminate the thread. If the thread is terminated, all object references are reclaimed during garbage collection.

When the change was made, the problem was solved immediately.

---

This article is from Zhu Kunrong's WeChat public account "Malt Bread", the public account id "darkjune_think"

Developers / sci-fi enthusiasts / hardcore console players / amateur translators, please indicate.

Communication Email: [e-mail]