Remember a memory leak analysis of a certain risk control management system in .NET

One: background

1. Tell a story

In the middle of last month, a friend on the planet came to me on WeChat, saying that his program would continue to grow slowly while it was running and could not be released. How to solve it?

Yes, it seems that the planet has to be handled well! ! ! 😂😂😂 Anyway, talk to windbg first.

Two: Windbg analysis

1. Empirical reasoning

From the screenshot of my friend, there are a lot of byte[] with 8216 bytes. What does this mean? Friends who follow this series should know that there is a a tertiary hospital with a skyrocketing memory. There is also size= (8216-24=8192) . His problem is that the OraBuf in the sdk is out when reading a large field in Oracle. The problem, in other words, this must be something created by the pool object in the underlying or third-party library. Next, look at the 16181ed342fcc3 managed heap.

2. View the managed heap

0:000> !dumpheap -stat
Statistics:
00007ffe107248f0   483707     15478624 System.Threading.PreAllocatedOverlapped
00007ffe1079c160   483744     15479808 System.Threading.ThreadPoolBoundHandle
00007ffe1079cff8   483701     23217648 System.Threading._IOCompletionCallback
00007ffe106e7a90   483704     23217792 Microsoft.Win32.SafeHandles.SafeFileHandle
00007ffe1079b088   483703     30956992 System.IO.FileSystemWatcher+AsyncReadState
00007ffe1079ceb0   483707     34826904 System.Threading.OverlappedData
00007ffe1079ccb0   483707     34826904 System.Threading.ThreadPoolBoundHandleOverlapped
0000016c64651080   245652   1473128080      Free
00007ffe105abf30   488172   3977571092 System.Byte[]

After sweeping the hosting pile, byte[] , 06181ed342fd47 did not attract me, but was System.IO.FileSystemWatcher+AsyncReadState . After all System.IO.FileSystemWatcher by 06181ed342fd4c many times, and it has deeply penetrated my mind. . . After all, it is it that makes the program stuck and makes the handle burst high. . . This time 80% of it was to blame again, and it seems that there are still many programmers planted here.

In order to be rigorous, I still start with the largest System.Byte[] , size it by 06181ed342fd97 and then in totalsize . I will not post the ugly script, and directly upload the output of the script.

!dumpheap -mt 00007ffe105abf30
size=8216,count=483703,totalsize=3790M
size=8232,count=302,totalsize=2M
size=65560,count=6,totalsize=0M
size=131096,count=2,totalsize=0M
size=4120,count=11,totalsize=0M
size=56,count=301,totalsize=0M
size=88,count=186,totalsize=0M
size=848,count=16,totalsize=0M
size=152,count=85,totalsize=0M
size=46,count=242,totalsize=0M
size=279,count=38,totalsize=0M

!dumpheap -mt 00007ffe105abf30 -min 0n8216 -max 0n8216 -short

0000016c664277f0
0000016c66432a48
0000016c6648ef88
0000016c6649daa8
0000016c6649fb00
0000016c664a8b90
...

From the output result, size=8216 are byte[] 06181ed342fe0c of 06181ed342fe0a, and then the script also lists some address addresses with the size of !gcroot see the references of these addresses.

0:000> !gcroot 0000016c664277f0
HandleTable:
    0000016C65FC28C0 (async pinned handle)
    -> 0000016C6628DEB0 System.Threading.OverlappedData
    -> 0000016C664277F0 System.Byte[]

Found 1 unique roots (run '!gcroot -all' to see all roots).
0:000> !gcroot 0000016c667c80d0
HandleTable:
    0000016C65FB7920 (async pinned handle)
    -> 0000016C663260F8 System.Threading.OverlappedData
    -> 0000016C667C80D0 System.Byte[]

From the output, you can see that these byte[] are all async pinned byte[] filled for 06181ed342fe70 when asynchronous IO comes back. Next, let's see how to byte[] place defined as 8192 in the source code OverlappedData

If you know FileSystemWatcher, the reverse lookup chain is about OverlappedData -> ThreadPoolBoundHandleOverlapped -> System.IO.FileSystemWatcher+AsyncReadState -> Buffer[] , which involves the binding of ThreadPool and SafeHandle.

0:000> !do 0000016C663260F8
Name:        System.Threading.OverlappedData
MethodTable: 00007ffe1079ceb0
EEClass:     00007ffe107ac8d0
Size:        72(0x48) bytes
File:        C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Private.CoreLib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ffe106e3c08  40009ce        8  System.IAsyncResult  0 instance 0000000000000000 _asyncResult
00007ffe104a0c68  40009cf       10        System.Object  0 instance 0000016c66326140 _callback
00007ffe1079cb60  40009d0       18 ...eading.Overlapped  0 instance 0000016c663260b0 _overlapped
00007ffe104a0c68  40009d1       20        System.Object  0 instance 0000016c667c80d0 _userObject
00007ffe104af508  40009d2       28                  PTR  0 instance 00000171728f66e0 _pNativeOverlapped
00007ffe104aee60  40009d3       30        System.IntPtr  1 instance 0000000000000000 _eventHandle
00007ffe104ab258  40009d4       38         System.Int32  1 instance                0 _offsetLow
00007ffe104ab258  40009d5       3c         System.Int32  1 instance                0 _offsetHigh
0:000> !do 0000016c663260b0
Name:        System.Threading.ThreadPoolBoundHandleOverlapped
MethodTable: 00007ffe1079ccb0
EEClass:     00007ffe107ac858
Size:        72(0x48) bytes
File:        C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Private.CoreLib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ffe1079ceb0  40009d6        8 ...ng.OverlappedData  0 instance 0000016c663260f8 _overlappedData
00007ffe1079b818  40009c0       10 ...ompletionCallback  0 instance 0000016f661ab8a0 _userCallback
00007ffe104a0c68  40009c1       18        System.Object  0 instance 0000016c667ca0e8 _userState
00007ffe107248f0  40009c2       20 ...locatedOverlapped  0 instance 0000016c66326090 _preAllocated
00007ffe104af508  40009c3       30                  PTR  0 instance 00000171728f66e0 _nativeOverlapped
00007ffe1079c160  40009c4       28 ...adPoolBoundHandle  0 instance 0000000000000000 _boundHandle
00007ffe104a7238  40009c5       38       System.Boolean  1 instance                0 _completed
00007ffe1079b818  40009bf      738 ...ompletionCallback  0   static 0000016f661ab990 s_completionCallback
0:000> !do 0000016c667ca0e8
Name:        System.IO.FileSystemWatcher+AsyncReadState
MethodTable: 00007ffe1079b088
EEClass:     00007ffe107a9dc0
Size:        64(0x40) bytes
File:        C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.IO.FileSystem.Watcher.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ffe104ab258  400002b       30         System.Int32  1 instance                1 <Session>k__BackingField
00007ffe105abf30  400002c        8        System.Byte[]  0 instance 0000016c667c80d0 <Buffer>k__BackingField
00007ffe106e7a90  400002d       10 ...es.SafeFileHandle  0 instance 0000016c66326028 <DirectoryHandle>k__BackingField
00007ffe1079c160  400002e       18 ...adPoolBoundHandle  0 instance 0000016c66326058 <ThreadPoolBinding>k__BackingField
00007ffe107248f0  400002f       20 ...locatedOverlapped  0 instance 0000016c66326090 <PreAllocatedOverlapped>k__BackingField
00007ffe1079b8c8  4000030       28 ...eSystem.Watcher]]  0 instance 0000016c66326078 <WeakWatcher>k__BackingField

The above <Buffer>k__BackingField was originally thrown to OverlappedData as the buffer for asynchronous IO read and write, and then look at the source code of System.IO.FileSystemWatcher+AsyncReadState

With these principles after, then you can ask a friend whether there appsettings set reloadonchange=true situation, find a friend under the code, probably written as follows:

public object GetxxxFlag()
{
    string value = AppConfig.GetConfig("appsettings.json").GetValue("xxxx", "0");

    return new
    {
        state = 200,
        data = value
    };
}

public class AppConfig
{
    public static AppConfig GetConfig(string settingfile = "appsettings.json")
    {
        return new AppConfig(settingfile);
    }
}

public class AppConfig
{
    private AppConfig(string settingfile)
    {
        _config = new ConfigurationBuilder().AddJsonFile(settingfile, optional: true, reloadOnChange: true).Build();
        _settingfile = settingfile;
    }
}

From the logic of the source code, I guess my friend thought it was singletonized after marking the GetConfig new AppConfig(settingfile) again will not repeat the 06181ed342ffde, so the problem lies here.

But what is interesting is that the FileSystemWatcher previous two articles will cause the program to freeze, so why not this one? It just so happens that he didn't put the log file in the program root directory, otherwise. . . 😄😄😄, but never expected to escape the deadlock but never escape the soul torture of 8byte . . 😂😂😂

Three: Summary

In general, reloadOnChange: true must be cautious when setting 06181ed3430056. It may cause your program to freeze, handle leaks, memory leaks, and so on! ! ! I won't talk about the improvement plan, refer to my previous series of articles.