Go memory leak, is pprof enough?

Text｜Zhu Dejiang (GitHub ID: doujiang24)

Core developer of MOSN project, technical expert of Ant Group

Focus on the related work of cloud native gateway research and development

This article is 2651 words read 8 minutes

MOSN is a cloud-native network proxy platform mainly developed in Go language, and has a large-scale production application of hundreds of thousands of containers in Ant Group. In such large-scale applications, various memory problems are often encountered. Usually, the pprof heap profile can be very helpful to analyze the problem. However, sometimes pprof is not enough, and we need more suitable tools.

Part.1--Birth Certificate vs Temporary Residence Certificate

First of all, pprof is really easy to use, and the design and implementation are also very delicate. If you are interested, you can check this "Go language pprof heap profile implementation mechanism" [1].

Using pprof to analyze memory leaks is usually enough, but sometimes it's not enough.

Why is this? Because pprof only records the call stack when the memory object is created, and there is no reference relationship . That is to say, there is no way to know that the memory object is not freed because of who is referencing it. In this regard, my colleague Lie Yuan has a very vivid analogy, pprof can only see the birth certificate, but cannot check the temporary residence certificate.

Part.2--requires reference relationship

In some scenarios, we know where the leaked memory is applied for, but after looking through the code for a long time, we can't figure out why the memory is not released. For example, memory objects are passed through complex calls, or complex memory pool reuse mechanism, or passed to an unfamiliar third-party library, and there are unexpected uses in the third-party library...

In these cases, we have a very intuitive idea that we want to see the reference relationship of these memory objects.

Part.3--memory reference relationship flame graph

The flame graph of memory reference relationship is a visualization method of memory object reference relationship, which was first applied to OpenResty XRay products. This tool is indeed a memory analysis artifact. It has located memory problems for many customers. If you are interested, you can go to the OpenResty official blog [2].

The following figure is generated by a MOSN service, the bottom-up representation is the reference relationship chain from the GC root to the GC object, and the width represents the size of the object (including the sum of the sizes of the objects it references) .

With such visualization results, we can intuitively see the reference relationship of memory objects.

For example, the widest part of the following figure represents the cluster memory object referenced in the cluster_manager global variable in MOSN:

https://github.com/mosn/mosn/blob/aecc93c4b2b4801e7992387f245fe9eefa45733d/pkg/upstream/cluster/cluster_manager.go#L82

Part.4--implementation principle

Before generating a flame graph, first we need to extract two key pieces of information:

- the reference relationship between each memory object;

- The type of each memory object.

reference relationship

Obtaining the reference relationship is relatively simple. First, we can find all GC objects in the heap. Then traverse all the objects, and then combine the bitmap information to obtain other objects referenced by this object. The basic principle is similar to GC mark, although the implementation is very different, but because this is an offline tool, it can be implemented simply and rudely.

type deduction

As a compiled static language, Go language does not need to store type information for each memory object (with a little exception is the interface) . If it is a dynamically typed language, such as Lua, it will be much more convenient, each GC object stores the type of the object.

Therefore, it is quite troublesome to obtain the type of each object, and it is also the most time-intensive piece. Of course, there is still a solution. Simply put, reverse type deduction is performed, and the type information of the referenced memory object is deduced based on the type information of the known memory.

This piece is still relatively complicated. If you are interested, you can read the introduction of this "Go Language, How to Do Reverse Type Deduction" [3].

Generation process

With these two key information, the generation process is relatively clear as follows:

1. Get all memory objects, including type, size, and the reference relationship between them to form a graph;

2. Starting from the root object, traverse according to the level to form a tree (that is, the pruning process, each object can only be referenced once) ;

3. Dump the complete reference relationship of this tree as a backtrace, count is the total size of the current node (including all child nodes) , which is the width on the flame graph;

4. Generate svg from bt file, this step is the FlameGraph standard tool chain of brendangregg.

Part.5--How to use

This tool is based on Go's official viewcore improvements. However, since the Go official is not so enthusiastic about maintaining viewcore, the MOSN community first forked a copy and set up a mosn branch as the main branch maintained by the MOSN community.

Several bugfixes have been submitted to the Go official debug before, and we will submit this feature when we are free later.

So, use it like this:

 # 编译 mosn 维护的 viewcore
git clone [e-mail]:mosn/debug.git
cd debug/cmd/viewcore
go build .

# 假设已经有了一个 core 文件（CORE-FILE）
# 以及对应的可执行程序文件（BIN-FILE）
viewcore CORE-FILE --exe BIN-FILE objref ref.bt

# 下载 FlameGraph 工具
git clone [e-mail]:brendangregg/FlameGraph.git
../FlameGraph/stackcollapse-stap.pl ref.bt | ../FlameGraph/flamegraph.pl> ref.svg

# 浏览器打开 ref.svg 即可看到火焰图

If you encounter problems, you can always contact us or submit an issue ( https://github.com/mosn/mosn/issues ).

Of course, if you successfully locate a problem, you are welcome to share it with us, Let's have fun together!

MOSN user DingTalk group: 33547952

Related Links

[1] "Go language pprof heap profile implementation mechanism": https://uncledou.site/2022/go-pprof-heap/

[2] OpenResty official blog: https://blog.openresty.com.cn/cn/openresty-xray-case-yundun/

[3] "Go language, how to do reverse type derivation": https://uncledou.site/2022/go-type-derivation/

understand more…

MOSN Star ✨: https://github.com/mosn/mosn

Insert a piece of good news! 🤩

Friends who are interested in Go language development, welcome to participate in the recent hot GoCity project experience

Click here to view the demo video and get started quickly 🥳

Recommended reading of the week

Detailed explanation of MOSN reverse channel

Full analysis of Go native plugin usage problems

Go code city to the cloud--KusionStack practice

MOSN document usage guide

Welcome to scan the code to follow: