Remember a memory leak analysis of a certain foreign trade Web site in .NET

One: background

1. Tell a story

Last Thursday, a friend added wx to consult that his program memory has a certain degree of leakage and cannot be recycled by GC. Eventually, the machine's memory is exhausted, which is embarrassing.

After communication, this friend’s ability is still very good, and I have done a preliminary dump analysis and found that there are 10w+ byte[] arrays on the managed heap, and it takes up about 1.1G of memory. After extracting a few byte[] of 060b063c37924c, I found that If there is no reference, I can't investigate further. Although I know that the problem may be in byte[], I can't find any evidence. 😪😪😪

Now that I have found me so trusting, I have to make a relatively comprehensive output report, and I can't live up to everyone's trust. It's still the old rules and talk to windbg.

Two: windbg analysis

1. Troubleshoot the source of the leak

Old readers who have read my article should know that to troubleshoot this kind of memory leak, we must first find out whether it is managed or unmanaged, so that we can take corresponding countermeasures.

Next, use !address -summary look at the committed memory of the process.

||2:2:080> !address -summary

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             573        1`5c191000 (   5.439 GB)  95.19%    0.00%
MEM_IMAGE                              1115        0`0becf000 ( 190.809 MB)   3.26%    0.00%
MEM_MAPPED                               44        0`05a62000 (  90.383 MB)   1.54%    0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                201     7ffe`9252e000 ( 127.994 TB)          100.00%
MEM_COMMIT                             1477        0`d439f000 (   3.316 GB)  58.04%    0.00%
MEM_RESERVE                             255        0`99723000 (   2.398 GB)  41.96%    0.00%

Judging from the MEM_COMMIT indicator of Guaxiang: Currently only 3.3G of memory usage, to be honest, I generally recommend 5G+ as the lowest threshold for memory leak analysis. After all, the larger the memory, the easier the analysis. Next, let’s look at the managed heap. Memory usage.

||2:2:080> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x00000002b37c0c48
generation 1 starts at 0x00000002b3781000
generation 2 starts at 0x0000000000cc1000

------------------------------
GC Heap Size:            Size: 0xbd322bb0 (3174181808) bytes.

It can be seen that the current managed heap occupies 3174181808/1024/1024/1024= 2.95G , haha, seeing this number, I was ecstatic, and the problem on the managed heap is almost stable to me. . . After all, I haven't missed it yet, so I quickly check the managed heap to see where the problem is.

2. View the managed heap

To view the managed heap, you can use the !dumpheap -stat command. Below I will display Top10 Size

||2:2:080> !dumpheap -stat
Statistics:
              MT    Count    TotalSize Class Name
00007ffd7e130ab8   116201     13014512 Newtonsoft.Json.Linq.JProperty
00007ffdd775e560    66176     16411648 System.Data.SqlClient._SqlMetaData
00007ffddbcc9da8    68808     17814644 System.Int32[]
00007ffddbcaf788    14140     21568488 System.String[]
00007ffddac72958    50256     22916736 System.Net.Sockets.SocketAsyncEventArgs
00007ffd7deb64b0      369     62115984 System.Collections.Generic.Dictionary`2+Entry[[System.Reflection.ICustomAttributeProvider, mscorlib],[System.Type, mscorlib]][]
00007ffddbcc8610     8348    298313756 System.Char[]
00007ffddbcc74c0  1799807    489361500 System.String
000000000022e250   312151    855949918      Free
00007ffddbccc768   109156   1135674368 System.Byte[]

As you can see from the above output, the current champion is Byte[] , the second place is Free , and the search is 060b063c3793dd. There are some String Byte[] into the basic types of 060b063c3793de and String , the input-output ratio is not high, after all A large number of complex types contain String and Byte[] in their internal structure. For example, I believe there must be Byte[] inside MemoryStream, right, so let's put down the champion and explore the flowers for the time being, and take a look at the No. 1 or other complex types.

If your eyes are sharp, you will find that the number of Free is 31W+ . You definitely want to ask what does this mean? Yes, this shows that there 31W+ on the current managed heap. Its technical term is fragmentation. So this piece of information reveals that the current managed heap has relatively serious fragmentation. The next question is why? In most cases, the reason for this fragmentation is that there are many pinned objects on the managed heap. This kind of object can prevent the GC from moving it during collection. Over the long term, it will cause the managed heap to become fragmented, so find out this phenomenon. Solving the leakage problem is of great help.

In addition, dotmemory can be used here. Red represents pinned objects. There are a large number of red intervals visible to the naked eye, and the final fragmentation rate is 85%.

The next question is how to find these pinned objects. In fact, there is a GCHandles table in the CLR, which records these things.

3. View GCHandles

To find all pinned objects, you can use the !gchandles -stat command, the simplified output is as follows:

||2:2:080> !gchandles -stat
Statistics:
              MT    Count    TotalSize Class Name
00007ffddbcc88a0      278        26688 System.Threading.Thread
00007ffddbcb47a8     1309       209440 System.RuntimeType+RuntimeTypeCache
00007ffddbcc7b38      100       348384 System.Object[]
00007ffddbc94b60     9359       673848 System.Reflection.Emit.DynamicResolver
00007ffddb5b7b98    25369      2841328 System.Threading.OverlappedData
Total 36566 objects

Handles:
    Strong Handles:       174
    Pinned Handles:       15
    Async Pinned Handles: 25369
    Ref Count Handles:    1
    Weak Long Handles:    10681
    Weak Short Handles:   326

It can be seen from the hexagram that there is currently a column: Async Pinned Handles: 25369 , which means that there are currently 2.5w . This indicator is quite abnormal, and it can be seen that it echoes the System.Threading.OverlappedData With this idea in mind, you can look back at the managed heap. Is there a corresponding 2.5w complex type object that encapsulates asynchronous operations? Here I will top10 Size the managed heap of 060b063c3794cf.

||2:2:080> !dumpheap -stat
Statistics:
              MT    Count    TotalSize Class Name
00007ffd7e130ab8   116201     13014512 Newtonsoft.Json.Linq.JProperty
00007ffdd775e560    66176     16411648 System.Data.SqlClient._SqlMetaData
00007ffddbcc9da8    68808     17814644 System.Int32[]
00007ffddbcaf788    14140     21568488 System.String[]
00007ffddac72958    50256     22916736 System.Net.Sockets.SocketAsyncEventArgs
00007ffd7deb64b0      369     62115984 System.Collections.Generic.Dictionary`2+Entry[[System.Reflection.ICustomAttributeProvider, mscorlib],[System.Type, mscorlib]][]
00007ffddbcc8610     8348    298313756 System.Char[]
00007ffddbcc74c0  1799807    489361500 System.String
000000000022e250   312151    855949918      Free
00007ffddbccc768   109156   1135674368 System.Byte[]

With this preconceived idea, I'm sure you found the managed heap 50256 of System.Net.Sockets.SocketAsyncEventArgs , evidently this time a leak and Socket can not get away, the next you can check these SocketAsyncEventArgs in the end whom the quote?

4. View SocketAsyncEventArgs reference root

To view the reference root, first export a few addresses from SocketAsyncEventArgs.

||2:2:080> !dumpheap -mt 00007ffddac72958 0 0000000001000000
         Address               MT     Size
0000000000cc9dc0 00007ffddac72958      456     
0000000000ccc0d8 00007ffddac72958      456     
0000000000ccc358 00007ffddac72958      456     
0000000000cce670 00007ffddac72958      456     
0000000000cce8f0 00007ffddac72958      456     
0000000000cd0c08 00007ffddac72958      456     
0000000000cd0e88 00007ffddac72958      456     
0000000000cd31a0 00007ffddac72958      456     
0000000000cd3420 00007ffddac72958      456     
0000000000cd5738 00007ffddac72958      456     
0000000000cd59b8 00007ffddac72958      456     
0000000000cd7cd0 00007ffddac72958      456     

Then look at the reference roots of the first and second addresses.

||2:2:080> !gcroot 0000000000cc9dc0
Thread 86e4:
    0000000018ecec20 00007ffd7dff06b4 xxxHttpServer.DaemonThread`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].DaemonThreadStart()
        rbp+10: 0000000018ececb0
            ->  000000000102e8c8 xxxHttpServer.DaemonThread`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]]
            ->  00000000010313a8 xxxHttpServer.xxxHttpRequestServer`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]]
            ->  000000000105b330 xxxHttpServer.HttpSocketTokenPool`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]]
            ->  000000000105b348 System.Collections.Generic.Stack`1[[xxxHttpServer.HttpSocketToken`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]], xxxHttpServer]]
            ->  0000000010d36178 xxxHttpServer.HttpSocketToken`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]][]
            ->  0000000008c93588 xxxHttpServer.HttpSocketToken`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]]
            ->  0000000000cc9dc0 System.Net.Sockets.SocketAsyncEventArgs
||2:2:080> !gcroot 0000000000ccc0d8
Thread 86e4:
    0000000018ecec20 00007ffd7dff06b4 xxxHttpServer.DaemonThread`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].DaemonThreadStart()
        rbp+10: 0000000018ececb0
            ->  000000000102e8c8 xxxHttpServer.DaemonThread`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]]
            ->  00000000010313a8 xxxHttpServer.xxxHttpRequestServer`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]]
            ->  000000000105b330 xxxHttpServer.HttpSocketTokenPool`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]]
            ->  000000000105b348 System.Collections.Generic.Stack`1[[xxxHttpServer.HttpSocketToken`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]], xxxHttpServer]]
            ->  0000000010d36178 xxxHttpServer.HttpSocketToken`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]][]
            ->  0000000000ccc080 xxxHttpServer.HttpSocketToken`2[[xxx.xxx, xxx],[xxx.RequestInfo, xxx]]
            ->  0000000000ccc0d8 System.Net.Sockets.SocketAsyncEventArgs

From the output information, it seems that the program has built an HttpServer and an HttpSocketTokenPool pool. The curiosity is coming. Export this class to see how to write it?

5. Find the problem code

Still the old way, use !savemodule export the problem code, and then use ILSpy to decompile.

To be honest, the package of this pool is quite crude. Since SocketAsyncEventArgs has 5W+, I guess m_pool pool. To verify the idea, you can use windbg to dig it out.

As can be seen from the size in the figure, this pool has about 2.5w HttpSocket, which shows that the so-called Socket Pool is actually not encapsulated.

Three: Summary

If you want to encapsulate a Pool by yourself, you have to implement some complex logic, and it can't just be a PUSH and POP. . . Therefore, the optimization direction is also very clear. Find a way to control the pool and achieve the effect that the pool should achieve.

More high-quality dry goods: see my GitHub: dotnetfly