Use Windbg to quickly analyze application memory leaks

Windbg program debugging is a necessary skill for advanced development of .NET. It analyzes memory leaks, analyzes high CPU usage, analyzes thread blocking, analyzes memory objects, analyzes thread stacks, and Live Dedugging. These Debug fields can be said to be a combination of skills + scenario-based applications. If you only rely on Windbg a few simple commands without understanding the role of each command in the actual troubleshooting, it is meaningless. A worker must first sharpen his tools if he wants to do his job well. Let's start with the commonly used commands and examples.

Windbg common commands

First prepare a Dump file, it is recommended to use a 64-bit application. For example: w3wp process of 64-bit IIS application, 64-bit exe process can be used. If you grab a Dump file, it is very simple: Task Manager-Process-Right-click [Create Dump File].

Then download and install Windbg, download link: https://developer.microsoft.com/zh-cn/windows/downloads/windows-10-sdk, all the way to the next step, select [Debugging Tools for Windows].

After opening, press Ctrl+D to open the Dump file captured in the first step, and start the introduction of common commands today.

1. Load the SOS debugging extension dll.

.loadby sos clr

2. Command to set and reload the debugging symbol file, download some important .Net pdb files to the specified path, and load them into the Windbg debugging environment, so that we can see which line of the program went wrong and which line it ran to .

.symfix+ C:\symbols
.reload

3. Print the search path of the current debugging symbol file.

0:000> .sympath

4. Check the thread pool, analyze and confirm the CPU usage, which instructions can be used.

0:000> !threadpool
CPU utilization: 2%
Worker Thread: Total: 19 Running: 2 Idle: 17 MaxLimit: 32767 MinLimit: 4
Work Request in Queue: 0
--------------------------------------
Number of Timers: 2
--------------------------------------
Completion Port Thread:Total: 4 Free: 4 MaxFree: 8 CurrentLimit: 4 MaxLimit: 1000 MinLimit: 4

5. View the overall operation of the thread.

!threads

6. Query the call stack of the specified thread, such as thread 34.

 ~34s

!clrstack

7. Check the CPU resource consumption of threads.

!runaway

The first column is the thread number, the second column is the total CPU usage time

8. View the information of all objects on the current thread stack, Dump stack objects.

9. Query the information about the specified object in the memory Dump object.

!do

10. Query the information Dump Array of the specified array object in the memory.

!de

11. View the stack of the current thread and the variable information on each line of the stack.

!clrstack -a

12. Windbg attach process debugging, enable CLR exception capture, view exceptions, view the thread stack where exceptions are located, disable CLR exception debugging, and exit debugging.

sxe clr
g
!pe
!clrstack
sxd clr
qd

13. View the distribution of memory objects on the managed heap and the information of the three generations.

!eeheap -gc

14. View the Dll loaded on the managed heap.

!eeheap -loader

15. Query the total number of various objects in the memory and the total memory usage.

!dumpheap  -stat

16. Query the number and size of large objects in memory.

!dumpheap -stat  -mt -min 85000

17. View the instructions of the memory destruction queue.

!finalizequeue

18. Please enter the command to view the gcroot of the object 000000123557DFC0.

!gcroot 000000123557DFC0

19. Look at the instructions that the thread is blocking.

!syncblk

20. Commands to view all System.Net.Sockets.Socket object statistics in Dump.

!dumpheap -type System.Net.Sockets.Socket -stat

The above are the commonly used commands and instructions for Windbg. Next, I will share with you the use of Windbg to analyze the memory leak of .NET applications.

Use Windbg to analyze the memory leak of .NET applications

We started today’s sharing according to the following ideas:

1. Describe the background and phenomenon of the problem
2. Determine if the problem is a memory leak
3. Sort out problem analysis ideas
4. Hands-on analysis and solution
5. Summary review

Let me talk about the background of the problem

In the production environment IIS site, after running for a period of time, the memory of the w3wp process will rise to 2G, and the memory will not be released.

question confirmed

Open the performance counter, we focus on the changes in the performance counters related to the w3wp process of the IIS site over a period of time:

Among the performance counters: Three are very important:

• .NET CLR Memory/Gen 2 heap size
• .NET CLR Memory/Gen 1 heap size
• .NET CLR Memory/Gen 0 heap size

There are three generations of objects on the managed heap:

Generation 0:

This is the youngest generation, which contains short-lived objects. An example of a short-lived object is a temporary variable. Garbage collection most often occurs in this generation. The newly allocated objects constitute a new generation of objects and are implicitly collected by generation 0 unless they are large objects, in which case they will enter the large object heap in the second generation of collection. Most objects are reclaimed by garbage collection in generation 0 and will not be retained to the next generation.

Generation 1:

This generation contains short-lived objects and serves as a buffer between short-lived objects and long-lived objects.

Generation 2:

This generation contains long-lived objects. An example of a long-lived object is an object in a server application that contains static data that is active during the process.
When the conditions are met, garbage collection will occur on a specific generation. Reclaiming a certain generation means reclaiming the objects in this generation and all their younger generations. The second generation of garbage collection is also called full garbage collection FullGC, because it reclaims all objects on all generations (that is, all objects in the managed heap).

Survival and promotion:

Objects that are not collected in garbage collection are also called survivors and will be promoted to the next generation. Objects surviving in generation 0 garbage collection will be promoted to generation 1; objects surviving in generation 1 garbage collection will be promoted to generation 2; and objects surviving in generation 2 garbage collection will remain The second generation.
Through generation promotion, look at the survival time of the object!

Process/Private Bytes
Process/Virtual Bytes
.NET CLR Memory/# Bytes in all Heaps: The size of the CLR memory managed heap
.NET CLR Memory/Large Object Heap Size: The large object heap contains objects whose size is 85,000 bytes and more.

The increasing trend of the memory size of the managed heap overlaps the increasing trend of the large object heap. It can be preliminarily inferred that the increase of memory is related to the large object!

Sort out problem analysis ideas

Grab two or three dumps consecutively and at intervals, with an interval of half an hour or one hour between each dump, and compare the memory increments through multiple dumps.

Look at each dump in comparison:

• In the case of multi-core CPU, analyze the size of each GC managed heap! eeheap -gc
• Query the total number and total memory usage of various objects in memory! dumpheap --stat
• Query the number and size of large objects in memory! dumpheap -stat -mt -min 85000
• If a certain type or several types of objects always occupy a lot of memory, analyze such objects! dumpheap -mt *
• Take multiple samples to view the gcroot of the object in step 4! Gcroot addr

Break any chain in gcroot and release object references

Hands-on analysis and solution

In the case of multi-core CPU, analyze the size of each GC managed heap! eeheap -gc.

Query the total number of various objects in the memory and the total memory usage! dumpheap -stat.

Query the number and size of large objects in memory! dumpheap -stat -mt -min 85000.

If a certain type or several types of objects always occupy a lot of memory, analyze such objects! dumpheap -mt * -stat.

Large object string analysis, you can combine the above figure with the specific code, you can find that it is the user Session session data! At the same time, the Session session contains the permission data of the entire user!

Take multiple samples to view the gcroot !gcroot addr of the object in step 4.

Interrupt the connection reference of any object in gcroot to complete the release of the object reference.

Summarize

To analyze and solve the memory leak problem of .NET applications, the general routine is this:

• Describe the background and phenomenon of the problem
• Determine if the problem is a memory leak
• Sort out problem analysis ideas
• Hands-on analysis and solution
• Summary review

The detailed analysis steps are as follows:

• In the case of multi-core CPU, analyze the size of each GC managed heap! eeheap -gc
• Query the total number and total memory usage of various objects in memory! dumpheap --stat
• Query the number and size of large objects in memory! dumpheap -stat -mt -min 85000
• If a certain type or several types of objects always occupy a lot of memory, analyze such objects! dumpheap -mt *
• Take multiple samples to view the gcroot of the object in step 4! Gcroot addr
• Break any reference chain in gcroot and release object references

There are fixed methods and routines for the analysis of memory leaks in .NET applications. The entire analysis process requires in-depth understanding and familiarity with the code is very important. The above sharing hopes to be helpful to everyone's daily online problem analysis.

Microsoft Most Valuable Professional (MVP)

Microsoft's Most Valuable Expert is a global award granted by Microsoft to third-party technology professionals. For 28 years, technology community leaders around the world have won this award for sharing their expertise and experience in online and offline technology communities.

MVP is a rigorously selected team of experts. They represent the most skilled and intelligent people. They are experts who are passionate and helpful to the community. MVP is committed to helping others through speeches, forum questions and answers, creating websites, writing blogs, sharing videos, open source projects, organizing conferences, etc., and to help users in the Microsoft technology community use Microsoft technology to the greatest extent.
For more details, please visit the official website:
https://mvp.microsoft.com/zh-cn

---

Welcome to follow the Microsoft China MSDN subscription account for more latest releases!