golang series: waitgroup analysis

Summary

Golang provides concise go keywords to make it easier for developers to perform concurrent programming, and also provides a WaitGroup object to assist concurrency control. Today we will analyze the use of WaitGroup, and take a look at its underlying source code.

Use scenarios and methods of WaitGroup

When we have many tasks to be performed at the same time, if we don't need to care about the execution progress of each task, just use the go keyword directly.

If we need to be concerned about the completion of all tasks before running down, we need WaitGroup to block waiting for these concurrent tasks.

WaitGroup, as it literally means, is to wait for a group of goroutines to finish running. There are three main methods:

• Add(delta int): add the number of tasks
• Wait(): Block waiting for the completion of all tasks
• Done(): complete the task

The following is their specific usage, the specific functions are in the comments:

package main

import (
    "fmt"
    "sync"
    "time"
)

func worker(wg *sync.WaitGroup) {
    doSomething()
    wg.Done() // 2.1、完成任务
}

func main() {
    var wg sync.WaitGroup
    wg.Add(5) // 1、添加 5 个任务
    for i := 1; i <= 5; i++ {
        go worker(&wg) // 2、每个任务并发执行
    }
    wg.Wait() // 3、阻塞等待所有任务完成
}

WaitGroup source code analysis

The use of WaitGroup above is very simple. Next, we will analyze its source code in src/sync/waitgroup.go. First, the structure of WaitGroup:

type WaitGroup struct {
    noCopy noCopy
    state1 [3]uint32
}

noCopy

Among them, noCopy means WaitGroup is not copyable. So what is non-copying?

For example, when we define this non-copyable type for function parameters, developers can only pass function parameters through pointers. What are the benefits of specifying pointer passing?

The advantage is that if there are multiple functions that define this non-copyable parameter, then these multiple function parameters can share the same pointer variable to synchronize the execution results. And WaitGroup needs such constraints.

state1 field

Next, let's take a look at the state1 field of WaitGroup. state1 is a uint32 array containing the total number of counters, the number of waiters, and the sema semaphore.

Whenever a goroutine calls the Wait() method to block waiting, the number of waiters + 1 and then wait for the notification of the semaphore.

When we call the Add() method, the number of counters in state1 + 1.

When the Done() method is called, the number of counters will be -1.

Until counter == 0, the goroutines corresponding to the number of waiters can be called up through the semaphore, that is, the goroutines that have just blocked waiting are called up.

For the explanation of the , please refer to the important knowledge of 1611a9295789ee golang: related introduction in

PV primitive explanation:
The problem of synchronization and mutual exclusion between processes is handled by operating the semaphore S.
S>0: indicates that there are S resources available; S=0 indicates that no resources are available; the absolute value of S<0 indicates the number of processes in the waiting queue or linked list. The initial value of the semaphore S should be greater than or equal to zero.
P primitive: Means to apply for a resource, subtract 1 atomically from S. If S>=0 after subtracting 1, then the process continues to execute; if S<0 after subtracting 1, it means that no resources are available and you need to change yourself Block it up and put it on the waiting queue.
V primitive: It means to release a resource and add 1 to S atomically; if S>0 after adding 1, the process continues to execute; if S<=0 after adding 1, it means there is a waiting process on the waiting queue, and you need to change The first waiting process wakes up.

Here, the operating system can be understood as the runtime runtime of Go, and the process can be understood as the coroutine .

Source code explanation

Finally, let's go deep into the three methods of WaitGroup for source code analysis. You can continue to look down if you are interested, mainly the analysis and comments on the source code.

Add(delta int) method

func (wg *WaitGroup) Add(delta int) {
    statep, semap := wg.state()
    if race.Enabled { // 此处是 go 的竞争检测，可以不用关心
        _ = *statep
        if delta < 0 {
            race.ReleaseMerge(unsafe.Pointer(wg))
        }
        race.Disable()
        defer race.Enable()
    }
    state := atomic.AddUint64(statep, uint64(delta)<<32)
    v := int32(state >> 32) // 获取 counter
    w := uint32(state) // 获取 waiter
    if race.Enabled && delta > 0 && v == int32(delta) { // go 的竞争检测，可以不用关心
        race.Read(unsafe.Pointer(semap))
    }
    if v < 0 {
        panic("sync: negative WaitGroup counter")
    }
    if w != 0 && delta > 0 && v == int32(delta) {
        panic("sync: WaitGroup misuse: Add called concurrently with Wait")
    }
    if v > 0 || w == 0 { // counter > 0：还有任务在执行；waiter == 0 表示没有在阻塞等待的 goroutine
        return
    }
    if *statep != state {
        panic("sync: WaitGroup misuse: Add called concurrently with Wait")
    }
    // 执行到此处相当于 countr = 0，即所有的任务都已执行完，需要唤起等待的 goroutine了
    *statep = 0
    for ; w != 0; w-- {
        runtime_Semrelease(semap, false, 0)
    }
}

Done method

func (wg *WaitGroup) Done() {
    wg.Add(-1) // 直接调用 Add 方法 对 counter -1
}

Wait method

func (wg *WaitGroup) Wait() {
    statep, semap := wg.state()
    if race.Enabled { // go 的竞争检测，可以不用关心
        _ = *statep
        race.Disable()
    }
    for {
        state := atomic.LoadUint64(statep)
        v := int32(state >> 32)
        w := uint32(state)
        if v == 0 {
            // counter 为 0， 不需要再等待了。
            if race.Enabled {
                race.Enable()
                race.Acquire(unsafe.Pointer(wg))
            }
            return
        }
        // waiters 数目 +1.
        if atomic.CompareAndSwapUint64(statep, state, state+1) {
            if race.Enabled && w == 0 {
                race.Write(unsafe.Pointer(semap)) // go 的竞争检测，可以不用关心
            }
            runtime_Semacquire(semap) // 阻塞等待唤起
            if *statep != 0 {
                panic("sync: WaitGroup is reused before previous Wait has returned")
            }
            if race.Enabled {
                race.Enable()
                race.Acquire(unsafe.Pointer(wg))
            }
            return
        }
    }
}

From the source code of these methods, we can see that Go does not use locks such as mutex to modify field values, but uses atomic operations to modify them. This is supported on the underlying hardware, so the performance is better.

Summarize

WaitGroup is relatively simple, it is the maintenance of some count values and the blocking of goroutine. It is also simple to use. The three methods of Add, Done, and Wait often appear at the same time. I believe that everyone can get a general idea even if they go deep into the source code, this is a shame.

---

"read new technology" to follow more push articles.
you can, just like, leave a comment, share, thank you for your support!
Read new technology, read more new knowledge.