How to effectively generate Dump in NET program thousands of ways to die (Part 1)

One: background

I believe many people know that the task manager is used to capture dump. Although simple and crude, it cannot satisfy the countless ways of death of the program, such as:

• Memory expansion, program explosion
• The CPU bursts high and the program is exhausted
• The app is unresponsive and the user is furious
• Quit unexpectedly, just like life

Since the handwork is too weak, what good tools are there? In addition to adplus, this article recommends an artifact procdump , download address: https://docs.microsoft.com/zh-cn/sysinternals/downloads/procdump , it can also support Linux 😘😘😘, how to install it is not detailed Up.

Two: memory expansion, program explosion

I believe that many of my friends have encountered this situation of memory expansion. The most common case I have seen is to use a small static cache, and then intentionally or unintentionally forget to release it, causing the infinite accumulation to finally explode. Then how to use procdump to catch What?

In order to facilitate the demonstration, I first write an example of unlimited memory allocation.

        static void Main(string[] args)
        {
            List<string> list = new List<string>();

            for (int i = 0; i < int.MaxValue; i++)
            {
                list.Add(string.Join(",", Enumerable.Range(0, 10000)));
            }

            Console.ReadLine();
        }

After running the program, set procdump to automatically grab the full memory dump when the memory exceeds 1G, use the following command.

C:\Windows\system32>procdump  ConsoleApp2 -m 1024 -ma E:\net5\ConsoleApp1\ConsoleApp2\bin\Debug

ProcDump v10.0 - Sysinternals process dump utility
Copyright (C) 2009-2020 Mark Russinovich and Andrew Richards
Sysinternals - www.sysinternals.com

Process:               ConsoleApp2.exe (24112)
Process image:         E:\net5\ConsoleApp1\ConsoleApp2\bin\Debug\ConsoleApp2.exe
CPU threshold:         n/a
Performance counter:   n/a
Commit threshold:      >= 1024 MB
Threshold seconds:     10
Hung window check:     Disabled
Log debug strings:     Disabled
Exception monitor:     Disabled
Exception filter:      [Includes]
                       *
                       [Excludes]
Terminate monitor:     Disabled
Cloning type:          Disabled
Concurrent limit:      n/a
Avoid outage:          n/a
Number of dumps:       1
Dump folder:           E:\net5\ConsoleApp1\ConsoleApp2\bin\Debug\
Dump filename/mask:    PROCESSNAME_YYMMDD_HHMMSS
Queue to WER:          Disabled
Kill after dump:       Disabled


Press Ctrl-C to end monitoring without terminating the process.

[21:23:43] Commit:    1087Mb
[21:23:43] Dump 1 initiated: E:\net5\ConsoleApp1\ConsoleApp2\bin\Debug\ConsoleApp2.exe_210323_212343.dmp
[21:23:43] Dump 1 writing: Estimated dump file size is 1179 MB.
[21:23:44] Dump 1 complete: 1179 MB written in 1.3 seconds
[21:23:44] Dump count reached.

It can be seen from the last five lines that 1087M , and then take a look at it with windbg.

• To view the memory usage of the current process, use !address -summary

0:000> !address -summary

                                     
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free                                     63          b30b4000 (   2.798 GB)           69.94%
<unknown>                               228          48547000 (   1.130 GB)  93.99%   28.25%
Image                                   210           4115000 (  65.082 MB)   5.29%    1.59%
Stack                                    21            700000 (   7.000 MB)   0.57%    0.17%
Heap                                     12            170000 (   1.438 MB)   0.12%    0.04%
Other                                     7             5a000 ( 360.000 kB)   0.03%    0.01%
TEB                                       7             13000 (  76.000 kB)   0.01%    0.00%
PEB                                       1              3000 (  12.000 kB)   0.00%    0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE                             250          47121000 (   1.110 GB)  92.36%   27.76%
MEM_IMAGE                               217           411e000 (  65.117 MB)   5.29%    1.59%
MEM_MAPPED                               19           1cfd000 (  28.988 MB)   2.35%    0.71%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE                                 63          b30b4000 (   2.798 GB)           69.94%
MEM_COMMIT                              357          47f12000 (   1.124 GB)  93.49%   28.10%
MEM_RESERVE                             129           502a000 (  80.164 MB)   6.51%    1.96%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE                          177          437d5000 (   1.055 GB)  87.70%   26.36%
PAGE_EXECUTE_READ                        35           33c7000 (  51.777 MB)   4.21%    1.26%
PAGE_READONLY                            90            c41000 (  12.254 MB)   1.00%    0.30%
PAGE_WRITECOPY                           34            70b000 (   7.043 MB)   0.57%    0.17%
PAGE_READWRITE|PAGE_GUARD                14             23000 ( 140.000 kB)   0.01%    0.00%
PAGE_EXECUTE_READWRITE                    7              7000 (  28.000 kB)   0.00%    0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free                                        80010000          7f130000 (   1.986 GB)
<unknown>                                   438e1000           200f000 (  32.059 MB)
Image                                       660e0000            f55000 (  15.332 MB)
Stack                                         e00000             fd000 (1012.000 kB)
Heap                                          c97000             98000 ( 608.000 kB)
Other                                       ff2c0000             33000 ( 204.000 kB)
TEB                                           990000              3000 (  12.000 kB)
PEB                                           98d000              3000 (  12.000 kB)

See PAGE_READWRITE line (1.055 GB) above? It echoes the 1087M in the Console just now, no problem.

• To find large objects, use !dumpheap -stat -min 1024

||0:0:000> !dumpheap -stat -min 1024
Statistics:
      MT    Count    TotalSize Class Name
65d42788        2        13044 System.Object[]
65d42d74        2        98328 System.String[]
65d42c60       73      1082988 System.Char[]
65d424e4    11452   1119913984 System.String

From the last line of the output, it can be seen that there are more than 1w of System.String -type attribute can be added to filter out the character string >10k

0:000> !dumpheap -type System.String -min 10240
 Address       MT     Size
03c75568 65d424e4    97792     
03c8d378 65d424e4    97792    
4a855060 65d424e4    97792     

Statistics:
      MT    Count    TotalSize Class Name
65d424e4    11452   1119913984 System.String
Total 11452 objects

0:000> !gcroot 4a855060
Thread 36e4:
*** WARNING: Unable to verify checksum for ConsoleApp2.exe
    00b3f358 012108d1 ConsoleApp2.Program.Main(System.String[]) [E:\net5\ConsoleApp1\ConsoleApp2\Program.cs @ 18]
        ebp+18: 00b3f370
            ->  02c71fd8 System.Collections.Generic.List`1[[System.String, mscorlib]]
            ->  02cce2ec System.String[]
            ->  4a855060 System.String

Found 1 unique roots (run '!GCRoot -all' to see all roots).

From the last !gcroot , it is indeed held by the List of line 06077e47b59c3b, and it has been Program.cs:18

Three: CPU bursts high, the program is exhausted

Speaking of high CPU explosive case, I found more in on unmanaged heap, such as GC recovery, competition for locks, etc., very few people dumb enough in on the hosting layer cpu engage them.

By the way, there is a small trick to analyze the CPU burst, that is, to continuously capture dump snapshots and see the running status of the threads in the two dumps. At this time, it is very suitable for procdump. Let's look at the test code first.

    class Program
    {
        static void Main(string[] args)
        {
            Parallel.For(0, int.MaxValue, (i) =>
            {
                while (true)
                {

                }
            });

            Console.ReadLine();
        }
    }

Now I set grab dumps by more than 70% of the CPU in 5s continuously until 2 dumps.

C:\Windows\system32>procdump  ConsoleApp2 -s 5 -n 2 -c 70 E:\net5\ConsoleApp1\ConsoleApp2\bin\Debug

ProcDump v10.0 - Sysinternals process dump utility
Copyright (C) 2009-2020 Mark Russinovich and Andrew Richards
Sysinternals - www.sysinternals.com

Process:               ConsoleApp2.exe (22152)
Process image:         E:\net5\ConsoleApp1\ConsoleApp2\bin\Debug\ConsoleApp2.exe
CPU threshold:         >= 70% of system
Performance counter:   n/a
Commit threshold:      n/a
Threshold seconds:     5
Hung window check:     Disabled
Log debug strings:     Disabled
Exception monitor:     Disabled
Exception filter:      [Includes]
                       *
                       [Excludes]
Terminate monitor:     Disabled
Cloning type:          Disabled
Concurrent limit:      n/a
Avoid outage:          n/a
Number of dumps:       2
Dump folder:           E:\net5\ConsoleApp1\ConsoleApp2\bin\Debug\
Dump filename/mask:    PROCESSNAME_YYMMDD_HHMMSS
Queue to WER:          Disabled
Kill after dump:       Disabled


Press Ctrl-C to end monitoring without terminating the process.

[22:25:47] CPU: 95% 1s
[22:25:48] CPU: 100% 2s
[22:25:50] CPU: 96% 3s
[22:25:51] CPU: 98% 4s
[22:25:52] CPU: 99% 5s (Trigger)
[22:25:53] Dump 1 initiated: E:\net5\ConsoleApp1\ConsoleApp2\bin\Debug\ConsoleApp2.exe_210323_222553.dmp
[22:25:54] Dump 1 complete: 5 MB written in 0.3 seconds
[22:25:56] CPU: 88% 1s
[22:25:58] CPU: 93% 2s
[22:26:00] CPU: 89% 3s
[22:26:02] CPU: 89% 4s
[22:26:04] CPU: 95% 5s (Trigger)
[22:26:05] Dump 2 initiated: E:\net5\ConsoleApp1\ConsoleApp2\bin\Debug\ConsoleApp2.exe_210323_222605.dmp
[22:26:06] Dump 2 complete: 5 MB written in 0.4 seconds
[22:26:07] Dump count reached.

From the final output, we can see that the continuous 5s CPU grabbed more than 70% of the dumps, and there were 2 dumps in total.

Now that the dump is available, open it with two windbg instances to verify the generation time of the dump, as shown in the following figure:

As you can see from the figure, the generation time of the two dumps is 12s apart, and the following threads are found !runaway

• 14:2cb8
• 19:3f8c
• ...

It has been running for up to 10s. What does this mean? Explain that these two threads should be in an infinite loop somewhere. . . Right. . .

Cut to the 14th thread and look at the call stack !clrstack ConsoleApp2.Program+c.b__0_0(Int32) can't get out. . .

Four: Summary

It feels a bit long, so let's stop here first. If you are interested, you can pull down procdump and have a fun🤭.