Remember a Windbg analysis. NET A factory MES system memory leak analysis

One: Background

1. Storytelling

Last month, a friend added WeChat for help, saying that his program ran and the memory exploded, and he was looking for how to solve it. The screenshots are as follows:

Judging from the chat content, this friend is still under a lot of pressure. In other words, this seems to be the third MES system I have analyzed. It seems that .NET is a giant in traditional factories. . .

Without further ado, let's use Windbg to find out.

Two: Windbg analysis

1. Managed or unmanaged

!address -summary look at the commit memory of the process, use 061db933215759.

0:000> !address -summary

                                     
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             971          e7d6b000 (   3.622 GB)  95.24%   90.56%
MEM_IMAGE                              1175           ac5d000 ( 172.363 MB)   4.43%    4.21%
MEM_MAPPED                               34            d08000 (  13.031 MB)   0.33%    0.32%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_COMMIT                             1806          edfd9000 (   3.719 GB)  97.77%   92.97%
MEM_FREE                                190           c920000 ( 201.125 MB)            4.91%
MEM_RESERVE                             374           56f7000 (  86.965 MB)   2.23%    2.12%

...

It can be seen currently occupied memory is 3.79G , from memory address to see a 32bit program, the program looks at the brink of collapse Kazakhstan 😂😂😂, then we look managed heap memory usage, use !eeheap -gc command.

0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0xf35a90c0
generation 1 starts at 0xf33a1000
generation 2 starts at 0x01db1000
ephemeral segment allocation context: none
 segment     begin  allocated      size
 ...
 f7790000  f7791000  f8058854  0x8c7854(9205844)
f33a0000  f33a1000  f3ba6e84  0x805e84(8412804)
Large object heap starts at 0x02db1000
 segment     begin  allocated      size
02db0000  02db1000  0387e988  0xacd988(11327880)
Total Size:              Size: 0xdcab5ca8 (3702217896) bytes.
------------------------------
GC Heap Size:    Size: 0xdcab5ca8 (3702217896) bytes.

From the output information, the managed heap memory occupies 3.7G , which is a relatively simple managed memory leak problem.

2. Explore the managed heap

To view the managed heap is still very simple, first come with a general command !dumpheap -stat .

0:000> !dumpheap -stat
Statistics:
      MT    Count    TotalSize Class Name
...
04b045d0    67663     25711940 xxx.Product.Mes.DataStore.EF.MesDbContext
719f0100  3458387     41500644 System.Object
719f1b84   281492     42391384 System.Int32[]
0489adb0  2238394     44767880 xxx.Application.Features.FeatureChecker
71551e00  2238503     53724072 System.Collections.Generic.List`1[[System.String, mscorlib]]
07c473e0  5615923     67391076 System.Data.Entity.Core.Objects.Internal.ObjectQueryExecutionPlanFactory
07c68954  5683589     68203068 System.Data.Entity.Core.Common.Internal.Materialization.Translator
04c7e3a8  4042677     71990132 Castle.DynamicProxy.IInterceptor[]
014a80c0  3142755     80480594      Free
042ecd18  5869494     93911904 xxxx.Domain.Uow.UnitOfWorkInterceptor
096ed32c    67663     97164068 System.Collections.Generic.Dictionary`2+Entry[[System.Type, mscorlib],[System.Data.Entity.Internal.Linq.IInternalSetAdapter, EntityFramework]][]
0488edb0 12641117    151693404 xxx.Domain.Uow.AsyncLocalCurrentUnitOfWorkProvider
0488fa50 10769173    215383460 xxx.Domain.Uow.UnitOfWorkManager
07cc0fb0  5548261    355088704 System.Data.Entity.Core.Objects.EntitySqlQueryState
719efd60 11275964   1268805768 System.String

From the perspective of the hexagram, the bottom sinks are basically EF . Relatively speaking, strings are generally held by these EFs, and a very abnormal place was also found, that is, MesDbContext actually has more than 6w, see The appearance is a little abnormal. Next, I will select a few to check the references, and the output is probably as follows:

0:000> !gcroot 17d2e438
HandleTable:
    014313c8 (pinned handle)
    -> 02dd9020 System.Object[]
    -> 0260abf4 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Data.Entity.DbContext, EntityFramework],[System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, mscorlib],[EntityFramework.DynamicFilters.DynamicFilterParameters, EntityFramework.DynamicFilters]], mscorlib]]
    -> b96074a4 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Data.Entity.DbContext, EntityFramework],[System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, mscorlib],[EntityFramework.DynamicFilters.DynamicFilterParameters, EntityFramework.DynamicFilters]], mscorlib]]
    -> 02fcddb0 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Data.Entity.DbContext, EntityFramework],[System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, mscorlib],[EntityFramework.DynamicFilters.DynamicFilterParameters, EntityFramework.DynamicFilters]], mscorlib]][]
    -> b955eecc System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Data.Entity.DbContext, EntityFramework],[System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, mscorlib],[EntityFramework.DynamicFilters.DynamicFilterParameters, EntityFramework.DynamicFilters]], mscorlib]]
    -> 17d2e438 xxx.DataStore.EF.MesDbContext

From the reference chain, these MesDbContext are ConcurrentDictionary<DbContext,ConcurrentDictionary<string,DynamicFilterParameters>> . Next, you need to judge how big the size of this dictionary is. You can use the !objsize command.

0:000> !objsize 0260abf4
e06d7363 Exception in c:\mysymbols\SOS_x86_x86_4.7.3701.00.dll\5F4FF1AE6f0000\SOS_x86_x86_4.7.3701.00.dll.objsize debugger extension.
      PC: 757ea842  VA: 022ce8f4  R/W: 19930520  Parameter: 7b9bb528

0:000> !DumpObj /d 02fcddb0
Name:        System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Data.Entity.DbContext, EntityFramework],[System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, mscorlib],[EntityFramework.DynamicFilters.DynamicFilterParameters, EntityFramework.DynamicFilters]], mscorlib]][]
MethodTable: 0973cb60
EEClass:     715c4fc0
Size:        573440(0x8c000) bytes
Array:       Rank 1, Number of elements 143357, Type CLASS (Print Array)
Fields:
None

After a long wait, I finally reported an error, but you can also see that there are 14.3w records in this dictionary, and the next serious problem comes, is this ConcurrentDictionary defined by a friend or within the framework? So the next step is to find its belonging class?

3. Explore which class the dictionary belongs to

It is relatively troublesome to find dictionary. For this, I recorded an episode on the B station to talk about this. Interested friends can take a look.


https://b23.tv/Rq47Vxp

In summary, the overall idea is:

1. First find the address (address1) of 17d2e438(MesDbContext) in 0260abf4(dictionary).
2. Then find the address (address2) of this address (address1) from memory.

This address2 exists in the method body that refers to this dictionary, and then you can decompile the method body, check its EEClass, and finally find the class name to which it belongs.

Next we will fight.

1. Check the size of object[].

0:000> !do 02dd9020
Name:        System.Object[]
MethodTable: 719f0154
EEClass:     715c4fc0
Size:        65532(0xfffc) bytes
Array:       Rank 1, Number of elements 16380, Type CLASS (Print Array)
Fields:
None

2. find address1

Search the memory with s -d

0:000> s -d 02dd9020 L?0xfffc 0260abf4
02de11a4  0260abf4 0260ad04 0260ad2c 08320d20  ..`...`.,.`. .2.

This 02de11a4 is what I was looking for address1, here to explain a little, -d represent the search by 32bit, -q search by 64bit, L?0xfffc is object [] array size .

3. look for address2

Here, the address will be split into 02de11a4 = a4 11 de 02 to search, otherwise there will be pits.

0:000> s-b 0 L?0xffffffff a4 11 de 02
0695d2f9  a4 11 de 02 e8 be 14 f9-6b b9 18 3c 34 70 e8 bc  ........k..<4p..
09e9438b  a4 11 de 02 39 09 e8 9a-11 af 67 8b f0 a1 bc 11  ....9.....g.....

From the output, there are two code areas that use dict , because it is a full memory search, here is the last address2=09e9438b .

4. decompile address2

Use !U decompile, and then !name2ee + !dumpmd + !dumpclass .

0:000> !U 09e9438b
Normal JIT generated code
EntityFramework.DynamicFilters.DynamicFilterExtensions.GetOrCreateScopedFilterParameters(System.Data.Entity.DbContext, System.String)
Begin 09e94320, size 1e1
09e94320 55              push    ebp
...
09e9433a 8bf1            mov     esi,ecx
09e9433c b95088ea09      mov     ecx,9EA8850h (MT: EntityFramework.DynamicFilters.DynamicFilterExtensions+<>c__DisplayClass71_0)
09e94341 e882ed5af7      call    014430c8 (JitHelp: CORINFO_HELP_NEWSFAST)
09e94346 8bf8            mov     edi,eax
09e94348 8d5704          lea     edx,[edi+4]
09e9434b e800a5a568      call    clr!JIT_WriteBarrierESI (728ee850)

0:000> !name2ee *!EntityFramework.DynamicFilters.DynamicFilterExtensions.GetOrCreateScopedFilterParameters
Module:      0973aef4
Assembly:    EntityFramework.DynamicFilters.dll
Token:       0600005e
MethodDesc:  0973b8fc
Name:        EntityFramework.DynamicFilters.DynamicFilterExtensions.GetOrCreateScopedFilterParameters(System.Data.Entity.DbContext, System.String)
JITTED Code Address: 09e94320

0:000> !dumpmd 0973b8fc
Method Name:  EntityFramework.DynamicFilters.DynamicFilterExtensions.GetOrCreateScopedFilterParameters(System.Data.Entity.DbContext, System.String)
Class:        0974c7d8
MethodTable:  0973b938
mdToken:      0600005e
Module:       0973aef4
IsJitted:     yes
CodeAddr:     09e94320
Transparency: Critical

0:000> !dumpclass 0974c7d8
Class Name:      EntityFramework.DynamicFilters.DynamicFilterExtensions
mdToken:         02000006
File:            D:\xxx\Debug\EntityFramework.DynamicFilters.dll
Parent Class:    715415b0
Module:          0973aef4
Method Table:    0973b938
Vtable Slots:    4
Total Method Slots:  20
Class Attributes:    100181  Abstract, 
Transparency:        Critical
NumInstanceFields:   0
NumStaticFields:     5
      MT    Field   Offset                 Type VT     Attr    Value Name
0973bfcc  400000d        c ....DynamicFilters]]  0   static 0260a9d4 _GlobalParameterValues
0973c3f4  400000e       10 ...ers]], mscorlib]]  0   static 0260abf4 _ScopedParameterValues
70343c18  400000f       14 ...tring, mscorlib]]  0   static 0260ad04 _PreventDisabledFilterConditions
71a34804  4000010       43       System.Boolean  1   static        1 _Initialized
05ec9adc  4000011       18 ...rsion, mscorlib]]  0   static 0260ad2c _OracleInstanceVersions

I finally found it. It turned out to be the EntityFramework.DynamicFilters.DynamicFilterExtensions class at the bottom of EF. The exported source code is as follows:

The final step is to take 6w more MesDbContext and 14w + of _ScopedParameterValues dictionary and friends do the communication, friends found a solution.





Three: Summary

According to the information provided by my friends, I finally commented out MesDbContext solve the problem. I am not familiar with EF, and friends who understand can leave a message for analysis.