Operation and maintenance: Your JAVA service memory usage is too high, and it is only increasing! Alarmed, come pick up the pot

Like it first, then watch it, develop a good habit

One day, the operation and maintenance brother suddenly asked me: "Some of your JAVA services have too high memory usage and an alarm! It is not released after GC, and the memory only increases and does not decrease. Is it a memory leak?"

Then I hurriedly checked the monitoring, everything was normal. It has been several days since the last release. FULL GC did not exist once, and YoungGC, once every ten minutes, the heap was free.

Operation and maintenance: "Your service now has only 800M heap memory used, but this JAVA process already occupies 6G of memory. Is there a memory leak bug in your program!"

I didn't even think about it, and directly replied: "Impossible, our service is very stable, there will be no such problem!"

But after speaking, I still questioned myself: Is there really any bug? Is it a leak outside the heap? The thread is not destroyed? Caused a memory leak? ? ?

Then I added a sentence "calmly": "I'll go to the server first to see what's going on". It's not good to be beaten in the face. It's better not to fill up too much...

I quickly boarded the server and carefully checked various indicators, such as Heap/GC/Thread/Process, and found that everything was normal, and there was no sign of "leakage".

"Communication" with operation and maintenance

Our service is normal, all indicators are ok, what kind of memory only increases and does not decrease, where is it?

Operation and maintenance: Look at your JAVA service. The heap is now only 400MB used, but the process now occupies 6G of memory. Still saying there is no problem? The memory must be leaked. Connect the pot and go back to check the problem.

Then I pointed to the monitoring information and let the operation and maintenance watch: "Big brother, look at the monitoring history. The heap memory has reached 6G, but the GC is behind, no problem!"

Operation and maintenance: "Your memory has not been released after recycling. You see if the process Res is still 6G, there must be a problem."

I think this operation and maintenance is not a der. JVM GC recycling and process memory are not the same thing, but I still have to explain to him, otherwise the baba will be endless.

"JVM garbage collection is just a logical collection. What is recycled is only the logical heap area requested by the JVM, marking data as free and other operations, instead of calling free to return the memory to the operating system."

After the operation and maintenance paused for two seconds, suddenly his face turned and he started to laugh: "Ahem, I may not pay attention to this. You can tell me about the relationship between JVM memory management/reclamation and process memory."

Although I reject it in my heart, I can’t offend operation and maintenance if I offend anyone. Think about it, I’ll explain it to my eldest brother, "Improve feelings"

Operating system and JVM memory allocation

The automatic memory management of JVM actually only applies for a large block of memory from the operating system first, and then performs "automatic memory management" in this applied memory area. Before the object in JAVA is created, it will first divide a part of the large block of memory from this application for use by this object. During GC, only the data in the memory area where the object is located is cleared and marked as free.

Operation and maintenance: "It turns out that this is the case. According to your opinion, will the JVM not return the free memory recovered by the GC to the operating system?"

Why not return the memory to the operating system?

The JVM will still return the memory to the operating system, but because this cost is relatively high, it will not be carried out easily. Moreover, different garbage collectors have different memory allocation algorithms, and the cost of returning memory is also different.

For example, in the sweep algorithm (sweep), memory is allocated through the free-list algorithm. Simply put, the large memory area that has been applied for is divided into N small areas, and these areas are organized with the structure of the linked list, like this:

Each data area can hold N objects. After a GC, some objects will be recycled, but there are other surviving objects in this data area at this time. If you want to release the entire data area, it will definitely not work.

Therefore, the operation of returning memory to the operating system is not that simple, and the execution cost is too high. Naturally, the JVM will not return the memory after each GC.

How to return it?

Although the cost is high, the JVM still provides the function of returning memory. JVM provides -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio , to configure this return strategy.

• MinHeapFreeRatio means that when the size of the free area drops to this value, expansion will be carried out, and the upper limit of expansion is Xmx
• MaxHeapFreeRatio represents when the free area exceeds this value, it will "shrink", and the lower limit of the shrink is Xms

However, although there is this return function, because the cost is more expensive, when the JVM returns, it is returned in a linear increment, not all at once.

But, after actual measurement, the mechanism of returning memory is different in different garbage collectors and even in different JDK versions!

Different versions & garbage collectors behave differently

Here are the test results I ran before:

public static void main(String[] args) throws IOException, InterruptedException {
    List<Object> dataList = new ArrayList<>();
    for (int i = 0; i < 25; i++) {
        byte[] data = createData(1024 * 1024 * 40);// 40 MB
        dataList.add(data);
    }
    Thread.sleep(10000);
    dataList = null; // 待会 GC 直接回收
    for (int i = 0; i < 100; i++) {
        // 测试多次 GC
        System.gc();
        Thread.sleep(1000);
    }
    System.in.read();
}
public static byte[] createData(int size){
    byte[] data = new byte[size];
    for (int i = 0; i < size; i++) {
        data[i] = Byte.MAX_VALUE;
    }
    return data;
}

JAVA version	Garbage collector	VM Options	Can it be "returned"
JAVA 8	UseParallelGC(ParallerGC + ParallerOld)	-Xms100M -Xmx2G -XX:MaxHeapFreeRatio=40	no
JAVA 8	CMS+ParNew	-Xms100M -Xmx2G -XX:MaxHeapFreeRatio=40 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC	Yes
JAVA 8	UseG1GC(G1)	-Xms100M -Xmx2G -XX:MaxHeapFreeRatio=40 -XX:+UseG1GC	Yes
JAVA 11	UseG1GC(G1)	-Xms100M -Xmx2G -XX:MaxHeapFreeRatio=40	Yes
JAVA 16	UseZGC(ZGC)	-Xms100M -Xmx2G -XX:MaxHeapFreeRatio=40 -XX:+UseZGC	no

The test results refreshed my understanding. , The MaxHeapFreeRatio parameter does not seem to be of much use, no matter if I configure 40 or 90, the recovery ratio is very different from the actual result.

But in the documentation, that's not what it says...

And the result of ZGC is quite unexpected. JEP 351 mentions that ZGC will release unused memory, but it is not in the test result.

In addition to the above test results, there are some other statements , I have not tested them one by one.

1. -XX:-ShrinkHeapInSteps parameter after JAVA 9 allows the JVM to return the memory in a non-linear increment
2. G1 after JAVA 12, when the application is idle, the memory can be automatically returned

Therefore, the statement of the official document can only be used as a reference, and the JVM does not disclose the implementation details too much.

However, except for this "enthusiastic" operation and maintenance brother, the general people would not care about the mechanism of whether to return or not. I

And not to mention that the idle is automatically returned. What we hope is to allocate the maximum memory as soon as it starts to prevent it from affecting the service when it is running; therefore, the general Java program will also Xms and Xmx be equal in size to avoid this expansion operation.

Hearing this, the operation and maintenance man thoughtfully said: "So as long as I configure Xms and Xmx to be the same size, this JAVA process will occupy this size of memory as soon as it starts?"

I then replied: "No, even if your Xms6G starts, it will only take up the memory actually written, and there is a high probability that it will not reach 6G. There is also a little knowledge about the memory allocation of the operating system."

Xms6G, why is used only 200M after startup?

When the process applies for memory, it does not directly allocate physical memory, but allocates a virtual space. data is written from 160ac927022a00 to the real heap, the physical memory allocated through the Page Fault processing mechanism. That is, the process Res indicator we see.

It can be simply considered that the memory allocation of the operating system is "lazy", the allocation does not occur actual occupation, memory occupation occurs only when data is written, which affects Res.

Therefore, even if Xms6G configured, it will not directly occupy 6G memory after startup, but JVM will malloc 6G after startup, but the actual memory occupied depends on whether you have written data to this 6G memory area.

Operation and maintenance: "Fuck, there is such a thing as lazy allocation! Long knowledge"

Me: "Understand now, this memory situation is normal, and there is no problem with our service at all"

Operation and maintenance: "🐂🍺, I misunderstood, there is nothing wrong with your service"

Me: "Well, it's okay, then I'll go ahead (fishing)"

to sum up

For most server-side scenarios, the manual memory release operation of JVM is not required. As for whether the JVM returns the memory to the operating system, we don't care. And based on the test results above, the difference between different JAVA versions and different garbage collector versions is so big that there is no need to go into it.

In summary, although the JVM can release free memory to the operating system, it will not necessarily release it. The performance is different under different JAVA versions and different garbage collector versions. Just know that there is this mechanism.

reference

• https://docs.oracle.com/javase/10/gctuning/factors-affecting-garbage-collection-performance.htm#JSGCT-GUID-B0BFEFCB-F045-4105-BFA4-C97DE81DAC5B
• https://stackoverflow.com/questions/30458195/does-gc-release-back-memory-to-os
• "In-depth understanding of the Java virtual machine: JVM advanced features and best practices (2nd edition)"-Zhou Zhiming

Originality is not easy, unauthorized reprinting is prohibited. If my article is helpful to you, please like/favorite/follow to encourage and support it!