The realization principle of defer in GO

[TOC]

The realization principle of defer in GO

Let’s review the previous sharing and share some knowledge points about the channel

• Shared what is the channel in GO
• Detailed analysis of the underlying data structure of the channel
• How the channel is implemented in the GO source code
• The basic principle of Chan reading and writing
• What are the exceptions when the channel is closed, panic
• Simple application of select

If you are still a little interested in the chan . Share the principle of Chan implementation in GO

What is defer?

Let's take a look at what defer is

is a keyword in GO

This keyword, we generally use to release resources, it will be called before return

If there are multiple defer program, the calling sequence of defer is similar to stack , last in, first out LIFO, write here by the way

• Stack

Follow the last-in first-out principle

After entering the stack, exit the stack first

The one that enters the stack first, then pops the stack

• queue

Following first-in-first-out, we can imagine a one-way pipeline, entering from the left and exiting from the right

Come first, go out first

Come in later, go out later, no jump in line

defer implementation principle

Let's throw out a conclusion first, and get a little bottomed out first:

• defer is declared in the code. When compiling, a function called deferproc will be inserted. Insert a returned function before the function where the defer return , deferreturn

What is the specific defer , we are still the same, let’s take a look at the underlying data structure of defer

type _defer struct { structure in src/runtime/runtime2.go

// A _defer holds an entry on the list of deferred calls.
// If you add a field here, add code to clear it in freedefer and deferProcStack
// This struct must match the code in cmd/compile/internal/gc/reflect.go:deferstruct
// and cmd/compile/internal/gc/ssa.go:(*state).call.
// Some defers will be allocated on the stack and some on the heap.
// All defers are logically part of the stack, so write barriers to
// initialize them are not required. All defers must be manually scanned,
// and for heap defers, marked.
type _defer struct {
   siz     int32 // includes both arguments and results
   started bool
   heap    bool
   // openDefer indicates that this _defer is for a frame with open-coded
   // defers. We have only one defer record for the entire frame (which may
   // currently have 0, 1, or more defers active).
   openDefer bool
   sp        uintptr  // sp at time of defer
   pc        uintptr  // pc at time of defer
   fn        *funcval // can be nil for open-coded defers
   _panic    *_panic  // panic that is running defer
   link      *_defer

   // If openDefer is true, the fields below record values about the stack
   // frame and associated function that has the open-coded defer(s). sp
   // above will be the sp for the frame, and pc will be address of the
   // deferreturn call in the function.
   fd   unsafe.Pointer // funcdata for the function associated with the frame
   varp uintptr        // value of varp for the stack frame
   // framepc is the current pc associated with the stack frame. Together,
   // with sp above (which is the sp associated with the stack frame),
   // framepc/sp can be used as pc/sp pair to continue a stack trace via
   // gentraceback().
   framepc uintptr
}

_defer holds an entry in the delayed call list. Let's see what the parameters of the above data structure mean

tag	Description
siz	The memory size of the parameters and results of the defer function
fn	Functions that need to be delayed
_panic	defer's panic structure
link	The defer delay function in the same coroutine will be connected together by this pointer
heap	Whether to allocate on the heap
openDefer	Whether through open coding optimization
sp	Stack pointer (usually corresponds to assembly)
pc	Program counter

The defer keyword must be followed by a function, we must remember this

Through the description of the above parameters, we can know that defer is similar to the function, and it also has the following three parameters:

• Stack pointer SP
• Program counter PC
• The address of the function

But did we also find out that there is also a link member, and the defer delay function in the same coroutine will be connected by this pointer

This link pointer points to the defer singly linked list. Every time we declare a defer , the defer data will be inserted into the head of the singly linked list.

Then, when defer is executed, can we guess defer was obtained?

I mentioned earlier that defer is last-in, first-out. Of course, this principle is also followed here. defer for execution, it is taken from the head of the singly linked list.

Let's draw a picture

In the coroutine A, declare 2 defer , first declare defer test1()

Re-statement defer test2()

It can be seen that the defer declared later will be inserted into the head of the singly linked list, and the defer declared first will be sorted to the back.

When we fetch it, we keep fetching it and execute it until the singly linked list is empty.

we take a look at defer specific implementation

The source code file is in src/runtime/panic.go , check the function deferproc

// Create a new deferred function fn with siz bytes of arguments.
// The compiler turns a defer statement into a call to this.
//go:nosplit
func deferproc(siz int32, fn *funcval) { // arguments of fn follow fn
   gp := getg()
   if gp.m.curg != gp {
      // go code on the system stack can't defer
      throw("defer on system stack")
   }

   // the arguments of fn are in a perilous state. The stack map
   // for deferproc does not describe them. So we can't let garbage
   // collection or stack copying trigger until we've copied them out
   // to somewhere safe. The memmove below does that.
   // Until the copy completes, we can only call nosplit routines.
   sp := getcallersp()
   argp := uintptr(unsafe.Pointer(&fn)) + unsafe.Sizeof(fn)
   callerpc := getcallerpc()

   d := newdefer(siz)
   if d._panic != nil {
      throw("deferproc: d.panic != nil after newdefer")
   }
   d.link = gp._defer
   gp._defer = d
   d.fn = fn
   d.pc = callerpc
   d.sp = sp
   switch siz {
   case 0:
      // Do nothing.
   case sys.PtrSize:
      *(*uintptr)(deferArgs(d)) = *(*uintptr)(unsafe.Pointer(argp))
   default:
      memmove(deferArgs(d), unsafe.Pointer(argp), uintptr(siz))
   }

   // deferproc returns 0 normally.
   // a deferred func that stops a panic
   // makes the deferproc return 1.
   // the code the compiler generates always
   // checks the return value and jumps to the
   // end of the function if deferproc returns != 0.
   return0()
   // No code can go here - the C return register has
   // been set and must not be clobbered.
}

The role of deferproc

Create a new deferred function fn , the parameter is siz byte, the compiler will convert a deferred statement into a call this

getcallersp()：

Get deferproc before rsp value of the register, the way to achieve all platforms are the same

//go:noescape
func getcallersp() uintptr // implemented as an intrinsic on all platforms

callerpc := getcallerpc()：

After getting rsp here, store it in callerpc , here is to call the next instruction of deferproc

d := newdefer(siz)：

d := newdefer(siz) creates a new defer , the subsequent code is to assign values to the members of this structure defer

Let's take a look at the general process of deferproc

• Get deferproc value before the register rsp
• Use newdefer allocate a _defer structure object and place it at the head of the _defer
• Initialize the relevant member parameters of _defer
• return0

to take a look at newdefer source

The source code file is in src/runtime/panic.go , check the function newdefer

// Allocate a Defer, usually using per-P pool.
// Each defer must be released with freedefer.  The defer is not
// added to any defer chain yet.
//
// This must not grow the stack because there may be a frame without
// stack map information when this is called.
//
//go:nosplit
func newdefer(siz int32) *_defer {
    var d *_defer
    sc := deferclass(uintptr(siz))
    gp := getg()
    if sc < uintptr(len(p{}.deferpool)) {
        pp := gp.m.p.ptr()
        if len(pp.deferpool[sc]) == 0 && sched.deferpool[sc] != nil {
            // Take the slow path on the system stack so
            // we don't grow newdefer's stack.
            systemstack(func() {
                lock(&sched.deferlock)
                for len(pp.deferpool[sc]) < cap(pp.deferpool[sc])/2 && sched.deferpool[sc] != nil {
                    d := sched.deferpool[sc]
                    sched.deferpool[sc] = d.link
                    d.link = nil
                    pp.deferpool[sc] = append(pp.deferpool[sc], d)
                }
                unlock(&sched.deferlock)
            })
        }
        if n := len(pp.deferpool[sc]); n > 0 {
            d = pp.deferpool[sc][n-1]
            pp.deferpool[sc][n-1] = nil
            pp.deferpool[sc] = pp.deferpool[sc][:n-1]
        }
    }
    if d == nil {
        // Allocate new defer+args.
        systemstack(func() {
            total := roundupsize(totaldefersize(uintptr(siz)))
            d = (*_defer)(mallocgc(total, deferType, true))
        })
    }
    d.siz = siz
    d.heap = true
    return d
}

The role of newderfer

Usually use per-P pool, allocate a Defer

Each defer can be freely released. The current defer will not be added to any defer chain

getg()：

Get the structure pointer of the current coroutine

// getg returns the pointer to the current g.
// The compiler rewrites calls to this function into instructions
// that fetch the g directly (from TLS or from the dedicated register).
func getg() *g

pp := gp.m.p.ptr()：

Get the P in the current worker thread

Then get a part of the object from the global object pool and give it to P's pool

for len(pp.deferpool[sc]) < cap(pp.deferpool[sc])/2 && sched.deferpool[sc] != nil {
                    d := sched.deferpool[sc]
                    sched.deferpool[sc] = d.link
                    d.link = nil
                    pp.deferpool[sc] = append(pp.deferpool[sc], d)
                }

Click in to see the data structure of the pool. In fact, the members inside are the 060cea83ca263f pointers _defer

Among them, sched.deferpool[sc] is the global pool, and pp.deferpool[sc] is the local pool

mallocgc allocate space

If the above operation does not get the value of d, then directly use mallocgc to redistribute, and set the corresponding members siz and heap

if d == nil {
        // Allocate new defer+args.
        systemstack(func() {
            total := roundupsize(totaldefersize(uintptr(siz)))
            d = (*_defer)(mallocgc(total, deferType, true))
        })
    }
d.siz = siz
d.heap = true

mallocgc implemented in src/runtime/malloc.go . If you are interested, you can take a closer look at this one. Today we will not focus on this function.

// Allocate an object of size bytes.
// Small objects are allocated from the per-P cache's free lists.
// Large objects (> 32 kB) are allocated straight from the heap.
func mallocgc(size uintptr, typ *_type, needzero bool) unsafe.Pointer {}

Finally, let's take a look at return0

Finally, let's take a look at the result in the deferproc return0()

// return0 is a stub used to return 0 from deferproc.
// It is called at the very end of deferproc to signal
// the calling Go function that it should not jump
// to deferreturn.
// in asm_*.s
func return0()

return0 is the stub used to return 0 deferproc

It is called at the end of the deferproc Go that it should not jump to deferreturn .

Under normal circumstances return0 returns 0 normally

However, under abnormal conditions, the return0 function will return 1 , at this time GO will jump to execute deferreturn

Simply talk about deferreturn

deferreturn action is the situation defer inside the chain, the return of the corresponding buffer, or the corresponding space for GC recovered modulation

The rules of defer in GO

After analyzing the defer in GO above, let's now understand that the application of defer in GO needs to comply with 3 rules. Let's list:

• defer is called a delayed function. The parameters in the function are defer statement is declared.
• The delay function is executed according to last-in, first-out. I have mentioned it many times in the previous article. This impression should be very deep. The first defer will be executed, and the later defer be executed first.
• Delayed functions may affect the return value of the entire function

Let’s explain . Point 2 above should be easy to understand. The figure above also shows the order of execution.

First, let’s write a small demo

The parameters in the delay function have been determined when the defer

func main() {
   num := 1
   defer fmt.Println(num)

   num++

   return
}

Don’t guess, the result is 1 . Friends can copy the code and run it by themselves.

The third point is also a DEMO

Delayed functions may affect the return value of the entire function

func test3() (res int) {
   defer func() {
      res++
   }()

   return 1
}
func main() {

   fmt.Println(test3())

   return
}

The above code, our test3 function, we named it in advance, it should be 1

But in return , the order of execution is like this

res = 1

res++

So the result is 2

to sum up

• Shared what defer is
• A simple illustration of the stack and queue
• Defer's data structure and implementation principle, specific source code display
• 3 rules of defer in GO

Welcome to like, follow, favorite

Friends, your support and encouragement are my motivation to keep sharing and improve quality

Okay, that's it for this time, Next time we use GO to play the verification code

Technology is open, and our mindset should be more open. Embrace the change, live toward the sun, and work hard to move forward.

I’m little magic boy , welcome to like and follow the collection, see you next time~