[Go Advanced - Basic Features] defer

defer is a keyword we often use. It executes the incoming function before the current function returns. It is often used to close file descriptors, close database connections, and unlock resources.

scenes to be used

release resources

This is the most common use of defer, including releasing mutexes, closing file handles, closing network connections, closing pipes, and stopping timers, such as:

 m.mutex.Lock()
defer m.mutex.Unlock()

exception handling

The second important use of defer is to handle exceptions, and to deal with panic together with recover, so that the program can recover from exceptions. For example, the source code of the recovery middleware in the gin framework:

 return func(c *Context) {
    defer func() {
        if err := recover(); err != nil {
            // Check for a broken connection, as it is not really a
            // condition that warrants a panic stack trace.
            var brokenPipe bool
            if ne, ok := err.(*net.OpError); ok {
                if se, ok := ne.Err.(*os.SyscallError); ok {
                    if strings.Contains(strings.ToLower(se.Error()), "broken pipe") || strings.Contains(strings.ToLower(se.Error()), "connection reset by peer") {
                        brokenPipe = true
                    }
                }
            }
// ...

Modify named return value

 // $GOROOT/src/fmt/scan.go
func (s *ss) Token(skipSpace bool, f func(rune) bool) (tok []byte, err error) {
    defer func() {
        if e := recover(); e != nil {
            if se, ok := e.(scanError); ok {
                err = se.err
            } else {
                panic(e)
            }
        }
    }()
...
}                        

print debug info

 // $GOROOT/src/net/conf.go
func (c *conf) hostLookupOrder(r *Resolver, hostname string) (ret hostLookupOrder) {
    if c.dnsDebugLevel > 1 {
        defer func() {
            print("go package net: hostLookupOrder(", hostname, ") = ", ret.String(), "\n")
        }()
    }
    ...
}                        

code of conduct

The syntax of defer is very simple, but there are many derived usages, which are sometimes confusing. Here we summarize some basic rules for using defer.

Functions following different defer keywords within the same function are executed in reverse order

As we will discuss later, the defer function after the defer keyword needs to be registered in a deferred function stack (essentially a linked list), so following the LIFO rule of the stack, the functions behind multiple defers are executed in reverse order.

Function arguments following the defer keyword are precomputed when the defer keyword appears

Functions following the defer keyword are evaluated when they are registered on the deferred function stack. Here's an example:

 func test1() {
    for i := 0; i <= 3; i++ {
        defer func(n int) {
            fmt.Println(n)
        }(i)
    }
}

func test2() {
    for i := 0; i <= 3; i++ {
        defer func() {
            fmt.Println(i)
        }()
    }
}

func main() {
    test1()
    test2()
}

In test1, defer is followed by an anonymous function with one parameter. Whenever defer registers an anonymous function on the deferred function stack, the parameters of the anonymous function are evaluated. Therefore, the parameters of the anonymous function in the deferred function stack are 0, 1, 2, 3 in sequence, and the printed result is 3, 2, 1, 0.

In test2, defer is followed by an anonymous function that takes no arguments. When the code is executed, the function popped from the deferred stack will access the external variable i in the form of a closure, and the value of i has become 4 at this time, so the print result is 4, 4, 4, 4.

Implementation principle

data structure

 // src/runtime/runtime2.go
type _defer struct {
    siz       int32
    started   bool
    heap    bool
    openDefer bool
    sp        uintptr
    pc        uintptr
    fn        *funcval
    _panic    *_panic
    link      *_defer
}

• siz is the memory size of arguments and results;
• heap indicates whether the structure is stored in the heap;
• sp and pc represent the stack pointer and the caller's program counter, respectively;
• fn is the function passed in the defer keyword;
• _panic is the structure that triggers the delayed call and may be empty;
• openDefer indicates whether the current defer is optimized by open coding.

It can be seen that each _defer instance is actually an encapsulation of a function, and has the necessary information to execute the function, such as stack pointer and so on. Multiple _defer instances are connected by a pointer link to form a singly linked list, which is stored in the data structure of the current goroutine, and then fetched and executed one by one after the execution of the current function is completed.

 type g struct {
    // ...
    _defer    *_defer
    // ...
}

execution mechanism

The defer in the program will be processed in the intermediate code generation stage. According to different conditions, there will be three different mechanisms to process the keyword: heap allocation, stack allocation and open encoding. The early Go language will allocate the _defer structure on the heap, but the performance of this implementation is poor. In version 1.13, the structure allocated on the stack was introduced, which reduced the extra overhead by 30%, and then the open-based structure was introduced in 1.14. The encoded defer greatly improves the performance.

heap allocation

Heap allocation is the default solution. When using this solution, the defer keyword is converted into a runtime.deferproc function during compilation, and a runtime.deferreturn function is inserted at the end of all functions that call defer. In short, deferproc generates a _defer structure and inserts it into the linked list, and deferreturn takes out _defer and executes it.

Let's take a brief look at the source code:

 func deferproc(siz int32, fn *funcval) {
    sp := getcallersp()
    argp := uintptr(unsafe.Pointer(&fn)) + unsafe.Sizeof(fn)
    callerpc := getcallerpc()

    d := newdefer(siz)
    if d._panic != nil {
        throw("deferproc: d.panic != nil after newdefer")
    }
    d.fn = fn
    d.pc = callerpc
    d.sp = sp
    switch siz {
    case 0:
    case sys.PtrSize:
        *(*uintptr)(deferArgs(d)) = *(*uintptr)(unsafe.Pointer(argp))
    default:
        memmove(deferArgs(d), unsafe.Pointer(argp), uintptr(siz))
    }

    return0()
}

deferproc will create a new _defer structure for defer, set its function pointer fn, program counter pc and stack pointer sp and copy the relevant parameters to the adjacent memory space.

The role of newdefer is to obtain the _defer structure, which contains three paths:

1. Take the structure from the scheduler's delayed call buffer pool sched.deferpool and append the structure to the current Goroutine's buffer pool;
2. Get the structure from the goroutine's delayed call buffer pool pp.deferpool;
3. Create a new structure on the heap via runtime.mallocgc.

No matter which method is used, after the _defer structure is obtained, it will be appended to the top of the _defer linked list of the Goroutine.

stack allocation

The stack allocation defer is introduced to improve the memory usage efficiency of the heap allocation defer. When the defer keyword is executed at most once in the function, the compiler will compile the defer into the deferprocStack function and allocate the _defer structure to the stack.

The _defer structure is already created during compilation, so deferprocStack just needs to set some uninitialized fields, and then append the _defer on the stack to the linked list.

 func deferprocStack(d *_defer) {
    gp := getg()
    d.started = false
    d.heap = false
    d.openDefer = false
    d.sp = getcallersp()
    d.pc = getcallerpc()
    d.framepc = 0
    d.varp = 0
    *(*uintptr)(unsafe.Pointer(&d._panic)) = 0
    *(*uintptr)(unsafe.Pointer(&d.fd)) = 0
    *(*uintptr)(unsafe.Pointer(&d.link)) = uintptr(unsafe.Pointer(gp._defer))
    *(*uintptr)(unsafe.Pointer(&gp._defer)) = uintptr(unsafe.Pointer(d))

    return0()
}

open coding

Whether it is a heap-allocated defer or a stack-allocated defer, the compiler can only convert the defer into the corresponding function that creates the _defer structure, and finally retrieve the structure through the deferreturn function and execute it. If the compiler is not so troublesome and directly converts the defer statement into the corresponding code and inserts it at the end of the function, can it save a lot of steps and improve storage efficiency and performance? Open coding uses this line of thinking.

But not all cases can use the open encoding method, in the following scenarios defer statement cannot be processed as open encoding type:

• Compiler with compiler optimizations disabled;
• defer appears in the loop;
• There are more than 8 defers in a single function;
• The number of return statements in a single function multiplied by the number of defer statements exceeds 15.