Have you heard of doing GC on Go map?

The map structure in Golang does not actually delete key-value pairs when deleting key-value pairs, but marks them. So with more and more key-value pairs, will it cause a lot of waste of memory?

First of all, the answer is yes, it is likely to lead to OOM, and there is another discussion about this: https://github.com/golang/go/issues/20135 . The rough meaning is that in the very large map , the delete operation does not really release the memory and may cause memory OOM.

So the general approach: reconstruct map . The container assembly of go-zero is built into safemap safemap can avoid this situation to a certain extent.

First of all, let's take a look at how the go provided by map is deleted?

Native map deletion

1  package main
2
3  func main() {
4      m := make(map[int]string, 9)
5      m[1] = "hello"
6      m[2] = "world"
7      m[3] = "go"
8
9      v, ok := m[1]
10     _, _ = fn(v, ok)
11
12     delete(m, 1)
13  }
14
15 func fn(v string, ok bool) (string, bool) {
16     return v, ok
17 }

The test code is as above, we can pass go tool compile -S -N -l testmap.go | grep "CALL" :

0x0071 00113 (test/testmap.go:4)        CALL    runtime.makemap(SB)
0x0099 00153 (test/testmap.go:5)        CALL    runtime.mapassign_fast64(SB)
0x00ea 00234 (test/testmap.go:6)        CALL    runtime.mapassign_fast64(SB)
0x013b 00315 (test/testmap.go:7)        CALL    runtime.mapassign_fast64(SB)
0x0194 00404 (test/testmap.go:9)        CALL    runtime.mapaccess2_fast64(SB)
0x01f1 00497 (test/testmap.go:10)       CALL    "".fn(SB)
0x0214 00532 (test/testmap.go:12)       CALL    runtime.mapdelete_fast64(SB)
0x0230 00560 (test/testmap.go:7)        CALL    runtime.gcWriteBarrier(SB)
0x0241 00577 (test/testmap.go:6)        CALL    runtime.gcWriteBarrier(SB)
0x0252 00594 (test/testmap.go:5)        CALL    runtime.gcWriteBarrier(SB)
0x025c 00604 (test/testmap.go:3)        CALL    runtime.morestack_noctxt(SB)

delete line 12, and the actual execution is runtime.mapdelete_fast64 .

These are the type of function parameters specific int64 , mapdelete_fast64 with the original delete operating the same, so we take a look mapdelete .

mapdelete

long picture warning! ! !

The general code analysis is as above, and the specific code is left for everyone to read. In fact, the general process:

1. Write protection to prevent concurrent writing
2. Query whether key to be deleted exists
3. If it exists, mark its mark for deletion
4. count--

So if you delete key in a large area, the actual map stored in key will not be deleted, but the current key status is marked as empty .

In fact, the starting point is mysql of the mark of 060eff21994863, which prevents the subsequent key , eliminating the need for expansion and contraction operations.

But this is not appropriate for some scenarios. If the developer will not insert the same key in the future, it is likely to cause OOM .

So in response to the above situation, go-zero developed safemap . Let's take a look at how safemap avoids this problem?

safemap

Analyze the reason for this design directly from the operation safemap

1. Preset a delete threshold , if triggered, it will be placed in a new preset newmap
2. Two map are a whole, so key can only keep one copy

So why set two map very clear:

1. dirtyOld main storage body. If the delete reaches the threshold, the migration will be triggered.
2. dirtyNew used as a temporary storage body, and will store part of key/value

So during the migration operation, what we need to do is: the original dirtyOld , the stored key/value is restored to dirtyNew through for-range, and then dirtyNew points to dirtyOld .

There may be questions: doesn't mean that key/value not deleted, but tophash=empty

In fact, for-range process, will filter out tophash <= emptyOne Key of

In this way, unnecessary keys will not be added to dirtyNew , which will not affect dirtyOld .

This is actually the concept of old and new generations of garbage collection.

For more implementation details, you can view the source code!

project address

https://github.com/tal-tech/go-zero

Welcome to use go-zero and star support us!

WeChat Exchange Group

Follow the " practice " public communication group get the QR code of the community group.