Why are Go maps and slices not thread-safe?

Hello everyone, I am fried fish.

When you first enter the door of the Go language, many small partners will quickly become proficient in Go in 3 days, 5 days to get started with the project, 14 days to go online for business iteration, 21 days to troubleshoot and locate problems, and bring in a reflection report by the way.

One of the most common elementary mistakes, one of the favorite questions in Go interviews:

(Questions from readers)

Why in Go language, map and slice do not support concurrent reading and writing, that is, they are not thread-safe, why not?

After seeing the tricks, we will start to discuss how to make the two of them "enemy" support concurrent reading and writing?

Today we will take this article to reason, understand its causes and consequences, and learn to understand the Go language together.

Non-thread safe example

slice

We use multiple goroutines to operate on a variable of type slice and see how the result will change.

as follows:

func main() {
 var s []string
 for i := 0; i < 9999; i++ {
  go func() {
   s = append(s, "脑子进煎鱼了")
  }()
 }

 fmt.Printf("进了 %d 只煎鱼", len(s))
}

Output result:

// 第一次执行
进了 5790 只煎鱼
// 第二次执行
进了 7370 只煎鱼
// 第三次执行
进了 6792 只煎鱼

You will find that no matter how many times you execute it, the probability of each output value will not be the same. That is, the value added to the slice is overwritten.

Therefore, the number added in the loop is not equal to the final value. And in this case, no error will be reported, which is an implicit problem with a low occurrence rate.

The main reason for this is that there is a problem with the program logic itself, and when the same index bit is read at the same time, the overwriting will naturally occur.

map

The same applies to map. Repeatedly write to a variable of type map.

as follows:

func main() {
 s := make(map[string]string)
 for i := 0; i < 99; i++ {
  go func() {
   s["煎鱼"] = "吸鱼"
  }()
 }

 fmt.Printf("进了 %d 只煎鱼", len(s))
}

Output result:

fatal error: concurrent map writes

goroutine 18 [running]:
runtime.throw(0x10cb861, 0x15)
        /usr/local/Cellar/go/1.16.2/libexec/src/runtime/panic.go:1117 +0x72 fp=0xc00002e738 sp=0xc00002e708 pc=0x1032472
runtime.mapassign_faststr(0x10b3360, 0xc0000a2180, 0x10c91da, 0x6, 0x0)
        /usr/local/Cellar/go/1.16.2/libexec/src/runtime/map_faststr.go:211 +0x3f1 fp=0xc00002e7a0 sp=0xc00002e738 pc=0x1011a71
main.main.func1(0xc0000a2180)
        /Users/eddycjy/go-application/awesomeProject/main.go:9 +0x4c fp=0xc00002e7d8 sp=0xc00002e7a0 pc=0x10a474c
runtime.goexit()
        /usr/local/Cellar/go/1.16.2/libexec/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00002e7e0 sp=0xc00002e7d8 pc=0x1063fe1
created by main.main
        /Users/eddycjy/go-application/awesomeProject/main.go:8 +0x55

Good guy, the program will report an error directly when it runs. And it is a fatal error caused by the Go source code calling the throw method, which means that the Go process will be interrupted.

I have to say, the error message fatal error: concurrent map writes I have a friend who has seen it dozens of times, different groups, different people...

It is an implicit problem of Nikkei.

How to support concurrent reading and writing

Lock map

In fact, we still have the appeal of concurrently reading and writing maps (determined by program logic), because the goroutine in Go language is so convenient.

When writing a crawler task, basically multiple goroutines are used, and the data is obtained and then written to a map or slice.

Go official provides a simple and convenient way to achieve in Go maps in action:

var counter = struct{
    sync.RWMutex
    m map[string]int
}{m: make(map[string]int)}

This statement declares a variable, which is an anonymous structure (struct) body, including a native and an embedded read-write lock sync.RWMutex .

To read data from the variable, call the read lock:

counter.RLock()
n := counter.m["煎鱼"]
counter.RUnlock()
fmt.Println("煎鱼:", n)

To write data to the variable, call the write lock:

counter.Lock()
counter.m["煎鱼"]++
counter.Unlock()

This is one of the most common ways that Map supports concurrent reading and writing.

sync.Map

Preface

Although there is a minimalist solution of Map+Mutex, there are still some problems. That is, when the amount of data in the map is very large, only one lock (Mutex) is very scary. One lock will cause a lot of contention for locks, leading to various conflicts and low performance.

The common solution is fragmentation, a large map is divided into multiple intervals, and each interval uses multiple locks, so that the granularity of the sub-locks is greatly reduced. However, the implementation of this scheme is very complicated and error-prone. Therefore, the Go team has no recommendation until the comparison, but has adopted other solutions.

This solution is 0614d5b5e50f3f supported in sync.Map , which supports concurrent read and write map, which plays a supplementary role.

Specific introduction

sync.Map Go language supports concurrent read and write map, adopts the "space for time" mechanism, and redundant two data structures, namely: read and dirty, to reduce the impact of locking on performance:

type Map struct {
 mu Mutex
 read atomic.Value // readOnly
 dirty map[interface{}]*entry
 misses int
}

It is specially designed for the append-only scene, which is suitable for scenes with more reading and less writing. This is one of his advantages.

If there are many write/concurrency scenarios, the read map cache will become invalid, locks will be required, conflicts will increase, and performance will drop sharply. This is his major shortcoming.

The following common methods are provided:

func (m *Map) Delete(key interface{})
func (m *Map) Load(key interface{}) (value interface{}, ok bool)
func (m *Map) LoadAndDelete(key interface{}) (value interface{}, loaded bool)
func (m *Map) LoadOrStore(key, value interface{}) (actual interface{}, loaded bool)
func (m *Map) Range(f func(key, value interface{}) bool)
func (m *Map) Store(key, value interface{})

• Delete: Delete the value of a certain key.
• Load: Returns the value of the key stored in the map, if there is no value, returns nil. The ok result indicates whether a value was found in the map.
• LoadAndDelete: Delete the value of a key, and return the previous value if there is one.
• LoadOrStore: If it exists, return the existing value of the key. Otherwise, it stores and returns the given value. If the value is loaded, the result of the load is true, if it is stored, it is false.
• Range: Recursive call, call the closure function f in turn for each key and value in the map. If f returns false, stop the iteration.
• Store: Store and set the value of a key.

The actual running example is as follows:

var m sync.Map

func main() {
 //写入
 data := []string{"煎鱼", "咸鱼", "烤鱼", "蒸鱼"}
 for i := 0; i < 4; i++ {
  go func(i int) {
   m.Store(i, data[i])
  }(i)
 }
 time.Sleep(time.Second)

 //读取
 v, ok := m.Load(0)
 fmt.Printf("Load: %v, %v\n", v, ok)

 //删除
 m.Delete(1)

 //读或写
 v, ok = m.LoadOrStore(1, "吸鱼")
 fmt.Printf("LoadOrStore: %v, %v\n", v, ok)

 //遍历
 m.Range(func(key, value interface{}) bool {
  fmt.Printf("Range: %v, %v\n", key, value)
  return true
 })
}

Output result:

Load: 煎鱼, true
LoadOrStore: 吸鱼, false
Range: 0, 煎鱼
Range: 1, 吸鱼
Range: 3, 蒸鱼
Range: 2, 烤鱼

Why not support

For Go Slice, the main problem is index bit overwriting. This does not need to be entangled. It is bound to have obvious flaws in the programming of the program logic, so it is better to correct it by yourself.

But the Go map is different. Many people think it is supported by default. It is so common that one accidentally rolls over. So why is it that Go officially doesn't support it? Is it too complicated and the performance is too poor? Why on earth?

The reasons are as follows (via @go faq):

• Typical usage scenario: The typical usage scenario of map is that it does not require secure access from multiple goroutines.
• Atypical scenario (requires atomic operations): map may be part of some larger data structure or already synchronized calculation.
• Performance scenario considerations: If you only increase security for a few programs, all operations on the map will have to deal with mutex, which will reduce the performance of most programs.

In summary, after a long period of discussion, the Go official believes that Go map should be more suitable for typical usage scenarios, rather than for a small part of the situation, which caused most programs to pay a price (performance) and decided not to support it.

Summarize

In today's article, we gave a basic introduction to map and slice in the Go language, and simulated the scenario that does not support concurrent readers.

At the same time, it also describes the common ways to support concurrent reading and writing in the industry, and finally analyzes the reasons for not supporting it, so that we have a complete understanding of the entire cause and effect.

I don’t know if has encountered non-linear security issues in the Go language in your daily life. You are welcome to leave a message in the comment area and with you 1614d5b5e5138f!

If you have any questions, welcome feedback and exchanges in the comment area. The best relationship between . Your likes is the biggest motivation for the creation of fried fish

The article is continuously updated, and you can read it on search [the brain is fried fish], this article 1614d5b5e5141f GitHub github.com/eddycjy/blog has been included, welcome to Star to remind you.