It's about slicing, but it doesn't seem to be just slicing?

From the public Gopher refers to the north

I have always had a desire in my heart, wanting to shout "I Hu Hansan is back again", and now is the right time.

I was a bit hand-skilled before I officially started, and I haven't written a technical article for too long. I am always a little lazy. I have to say that persistence is indeed a difficult thing. If it wasn't for guilt and writing something to calm myself down a bit, I might still be lazy.

There is another interesting thing to share. The previous article on the public account only took nearly a month to think about, and only a day to write, but I usually think and write technical articles on average for a week. The result is not deceiving. The reading volume of the previous article exceeded one thousand for the first time. Sure enough, the depth of the reader's thinking is at least as much as a month. Old Xu admires it.

Slice the underlying structure

Transformation of slices and structures

Other things are not too much, we still return to the theme of this article. Before we formally understand the underlying structure of slices, let's look at a few lines of code.

type mySlice struct {
    data uintptr
    len  int
    cap  int
}

s := mySlice{}
fmt.Println(fmt.Sprintf("%+v", s))
// {data:0 len:0 cap:0}
s1 := make([]int, 10)
s1[2] = 2
fmt.Println(fmt.Sprintf("%+v, len(%d), cap(%d)", s1, len(s1), cap(s1))) // [0 0 2 0 0 0 0 0 0 0], len(10), cap(10)
s = *(*mySlice)(unsafe.Pointer(&s1))
fmt.Println(fmt.Sprintf("%+v", s)) // {data:824634515456 len:10 cap:10}
fmt.Printf("%p, %v\n", s1, unsafe.Pointer(s.data)) // 0xc0000c2000, 0xc0000c2000

In the above code, by obtaining the address of the slice and converting it to *mySlice , the length and capacity of the slice are successfully obtained. And something similar to a pointer. And this pointer points to the array that stores the real data, let's verify it below.

//Data强转为一个数组
s2 := (*[5]int)(unsafe.Pointer(s.data))
s3 := (*[10]int)(unsafe.Pointer(s.data))
// 修改数组中的数据后切片中对应位置的值也发生了变化
s2[4] = 4
fmt.Println(s1)  // [0 0 2 0 4 0 0 0 0 0]
fmt.Println(*s2) // [0 0 2 0 4]
fmt.Println(*s3) // [0 0 2 0 4 0 0 0 0 0]

At this point, the underlying structure of the slice is ready to come out, but in order to do further verification, we continue to test the process of turning the structure into a slice.

var (
    // 一个长度为5的数组
    dt [5]int
    s4 []int
)
s5 := mySlice{
    // 将数组地址赋值给data
    data: uintptr(unsafe.Pointer(&dt)),
    len:  2,
    cap:  5,
}
// 结构体强转为切片
s4 = *((*[]int)(unsafe.Pointer(&s5)))
fmt.Println(s4, len(s4), cap(s4)) // [0 0] 2 5
// 修改数组中的值， 切片内容也会发生变化
dt[1] = 3
fmt.Println(dt, s4) // [0 3 0 0 0] [0 3]

Through the above three pieces of code, we express the underlying structure of the slice more clearly in the form of a structure. As shown in the figure below, the first part (Data) is the address pointing to the array, the second part (Len) is the length of the slice that is the length of the array used, and the third part (Cap) is the length of the array.

Summary : A slice is a packaging of an array, and the bottom layer still uses an array to store data.

One more talk:

The reflect package uses the reflect.SliceHeader structure when operating the slice, see for details https://github.com/golang/go/blob/master/src/reflect/value.go#L2329

The runtime uses the slice structure when expanding the slice. For details, see https://github.com/golang/go/blob/master/src/runtime/slice.go#L12

unsafe digression

The use of the unsafe package is almost inseparable from the demo in the previous part. Of course, this article does not introduce the usage of this package, just as a digression to briefly look at why it is not safe.

func otherOP(a, b *int) {
    reflect.ValueOf(a)
    reflect.ValueOf(b)
}

var (
    a = new(int)
    b = new(int)
)
otherOP(a, b) // 如果注释此函数调用，最终输出结果会发生变化
*(*int)(unsafe.Pointer(uintptr(unsafe.Pointer(a)) + unsafe.Sizeof(int(*a)))) = 1
fmt.Println(*a, *b)

The output results of the above generation are inconsistent when whether to comment otherOP

When the variable escapes to the heap, the a and the variable b adjacent, so the value of the b variable can be set through the a variable address. When it did not escape to the heap, setting b did not take effect, so we could not know which piece of memory was modified at all. This uncertainty in Lao Xu's view is that we need to use this package carefully. s reason.

Supplementary explanation about the above demo:

1. reflect.ValueOf will call the underlying escapes method to ensure that the object escapes to the heap
2. Go uses a free linked list memory allocator divided by size and a multi-level cache, so a and b variables are likely to be allocated to a continuous memory space when the size is the same and the demo variables are less.

Create slice

There are four ways to create slices. The first is directly through var for variable declaration, the second is through type deduction, the third is make , and the fourth is created through slice expressions.

// 通过变量声明的方式创建
var a []int
// 类型推导
b := []int{1, 2, 3}
// make创建
c := make([]int, 2) // c := make([]int, 0, 5)
// 切片表达式
d := c[:3]

In the above examples, the first three are nothing particularly to say. Old Xu mainly introduces the fourth, and its related restrictions and precautions.

Simple slice expression

For strings, arrays, array pointers, and slices (slice pointers cannot use the following expressions), the following expressions can be used:

s[low:high] // 生成的切片长度为high-low

Through the above expressions, new substrings or slices can be created. Special attention is that when using this expression on a string, neither a string slice nor a byte slice is generated but a is generated. In addition, Lao Xu Go's string encoding problem that the string in Go stores utf8 byte slices, so we may encounter unexpected when we use this method to obtain special characters containing Chinese and other special characters. result. The correct way to get the substring should be to first convert to rune slices and then intercept.

The above expression can already be very convenient to create a new slice, but it is more convenient that low and high can also be omitted.

s[2:]  // same as s[2 : len(a)]
s[:3]  // same as s[0 : 3]
s[:]   // same as s[0 : len(a)]

subscript limit

Using slice expressions for different types, the value ranges of low and high For string and array / array pointer terms, low and high range of 0 <= low <= len(s) . For the purposes of sections, low and high the range of 0 <= low <= cap(s) . In the slice interview question series , it is the investigation of this knowledge point.

Slice capacity

The slice generated by the slice expression shares the underlying array, so the capacity of the slice is the length of the underlying array minus low . It can be inferred that the following code output results are 3 8 and 3 13 .

var (
    s1 [10]int
    s2 []int = make([]int, 10, 15)
)
s := s1[2:5]
fmt.Println(len(s), cap(s))
s = s2[2:5]
fmt.Println(len(s), cap(s))
return

Complete slice expression

To be honest, this method is really not commonly used. Although it can control the capacity of slices, Xu has not used it in actual development. The complete expression is as follows:

s[low : high : max]

There are several points to note about this expression:

• Only applicable to arrays, array pointers and slices. Not applicable to strings.
• Different from the simple slice expression, it can only ignore the low and the default value of the subscript is 0 after being ignored.
• Like a simple slice expression, the underlying array of slices generated by a complete slice expression is shared

subscript limit

For array/array pointers, the subscript value range is 0 <= low <= high <= max <= len(s) . For slices, the subscript value range is 0 <= low <= high <= cap(s) . In the slice interview question series two , it is the investigation of this knowledge point.

Slice capacity

As mentioned earlier, this slice expression can control the capacity of the slice. When low constant, the capacity of the slice can be changed by changing max within the allowable range. The capacity calculation method is max - low .

Slicing expansion

The growslice runtime/slice.go file implements the expansion logic of the slice. Before we formally analyze the internal logic, let's take a look at the function signature of growslice

type slice struct {
    array unsafe.Pointer
    len   int
    cap   int
}

func growslice(et *_type, old slice, cap int) slice

The first parameter _type is the runtime representation of the Go language type, which contains a lot of meta-information, such as type size, alignment, and type.

The second parameter is the information of the slice to be expanded.

The third parameter is the actual required capacity, that is, the sum of the original capacity and the number of new elements, and Lao Xu referred to it as the required capacity for short. To make it easier to understand the meaning of the required capacity, let's first look at a piece of code:

s := []int{1,2,3} // 此时切片长度和容量均为3
s = append(s, 4) // 此时所需容量为3 + 1
s1 := []int{1,2,3} // 此时切片长度和容量均为3
s1 = append(s1, 4, 5, 6) // 此时所需容量为3 + 3

Expansion logic

With the above concepts, let's look at the slice expansion algorithm below:

The logic of the above figure is summarized as follows:

First, if the required capacity is greater than 2 times the current capacity, the new capacity is the required capacity.

Second, determine whether the current capacity is greater than 1024. If the current capacity is less than 1024, the new capacity is equal to 2 times the current capacity. If the current capacity is greater than or equal to 1024, the new capacity is cyclically increased by 1/4 times the new capacity until the new capacity is greater than or equal to the required capacity.

Old Xu here specially reminds that 0 is useful. At the beginning, Old Xu also felt that this logic was very redundant, but one day he suddenly realized that this was actually a judgment on plastic overflow. Because this issue is rarely considered in normal development, I was shocked for a while. Perhaps the code gap between us and the Great God is just a lack of judgment on overflow.

Another interesting thing is that the logic of slicing was not like this at the beginning. The logic is not complicated, and even those who are just getting started can write it without pressure. However, even such a simple logic has been iterated through multiple versions before it becomes what it is today.

One thing to say is that the expansion logic in 1.6 is not elegant in the eyes of Lao Xu. Thinking of this, a feeling of "I won" spontaneously arises. The happiness of programmers is so simple.

Calculate memory capacity

The expansion logic in the previous article is the ideal memory allocation capacity, but the real memory allocation is very complicated. In Go1.6, memory allocation for slice expansion is divided into four situations, namely the type size is 1 byte, the type size is the pointer size, the type size is the power of 2 and others. The memory allocation for slice expansion is slightly different in different Go versions. Here, we only introduce the memory allocation when the type size is 2 to the power of n in 1.16.

Directly on the code below:

var shift uintptr
if sys.PtrSize == 8 {
    // Mask shift for better code generation.
    // et.size = 1 << n
    // shift = n
    // &63是因为uint64中1最大左移63，再大就溢出了
    shift = uintptr(sys.Ctz64(uint64(et.size))) & 63
} else {
    shift = uintptr(sys.Ctz32(uint32(et.size))) & 31
}

In the above code, the size of the pointer is judged to distinguish whether it is currently running on a 32-bit or 64-bit platform. Ctz64 and Ctz32 functions calculate the number of the 0 for different types. And because the type size is 2 to the power of n, the number of 0s is n.

The type size is 2 to the power of n, then the type size must be 1 << n, so the number of the lowest bit 0 can be calculated to get the number of bits shifted to the left.

In the source code, the x&(x-1) == 0 expression is used to determine whether an unsigned integer is 2 to the power of n. If there is a similar logic in our usual development, please refer to the slice expansion source code to start the journey of loading.

Next is the logic to calculate the memory capacity:

capmem = roundupsize(uintptr(newcap) << shift)
newcap = int(capmem >> shift)

In combination with the foregoing, it is easy to know that uintptr(newcap) << shift can actually be understood as uintptr(newcap) * et.size , and capmem >> shift can be understood as capmem / et.size . uintptr(newcap) << shift is the most ideal required memory size, but the actual memory allocation cannot achieve the ideal condition due to memory alignment and other issues, so roundupsize , and finally the real capacity is calculated. With this understanding, we next focus on analyzing the roundupsize function.

func roundupsize(size uintptr) uintptr {
    if size < _MaxSmallSize {
        if size <= smallSizeMax-8 {
            return uintptr(class_to_size[size_to_class8[divRoundUp(size, smallSizeDiv)]])
        } else {
            return uintptr(class_to_size[size_to_class128[divRoundUp(size-smallSizeMax, largeSizeDiv)]])
        }
    }
    if size+_PageSize < size {
        return size
    }
    return alignUp(size, _PageSize)
}

The above function has many variables with unclear meanings, and Lao Xu will explain them one by one next.

_MaxSmallSize : The value is 32768 , which is 32kb in size. In Go, when the size of an object exceeds 32kb, the memory allocation strategy is different from when it is less than or equal to 32kB.

smallSizeMax : The value is 1024 bytes.

smallSizeDiv : Its value is 8 bytes.

largeSizeDiv : The value is 128 bytes.

_PageSize : 8192 bytes, which is 8kb in size. Go manages memory in pages, and each page is 8kb in size.

class_to_size : The memory allocation in Go will divide the memory into different memory block linked lists according to different spans (also known as memory size). When memory needs to be allocated, the free memory block is found according to the object size to match the most suitable span. There are a total of 67 spans in Go. class_to_size is an array with a length of 68, which records the 0 and these 67 spans respectively. For the source code, please refer to sruntime/izeclasses.go .

size_to_class8 : This is an array with a length of 129, representing the memory size range of 0~1024 bytes. An index i an example, the size of this target position m is i * smallSizeDiv , size_to_class8[i] value of class_to_size array span closest m subscripts.

size_to_class128 : This is an array with a length of 249, representing the memory size range of 1024~32768 bytes. An index i an example, this position of the object size m of smallSizeMax + i*largeSizeDiv , size_to_class128[i] value of class_to_size array span closest m subscripts.

divRoundUp : This function returns a/b rounded up to the nearest integer.

alignUp: alignUp(size, _PageSize) = _PageSize * divRoundUp(size, _PageSize)。

Finally, the logical expression for calculating the actual memory size required is as follows:

At this point, the core logic of slicing expansion has been analyzed. The reason why this article does not analyze the expansion logic for the type size of 1 byte, the type size of the pointer size, and other sizes is that the overall logic is not very different. In Lao Xu's view, the main purpose of the type size distinction in the source code is to reduce the division and multiplication operations as much as possible. Every time I read these excellent source code, Lao Xu calls the detail monster directly.

In order to deepen the impression, we use slice interview question series three to perform a calculation.

s3 := []int{1, 2}
s3 = append(s3, 3, 4, 5)
fmt.Println(cap(s3))

According to the previous knowledge, the required capacity is 5, and because the required capacity is greater than twice the current capacity, the new capacity is also 5.

And because the size of the int type is 8 (equal to the size of the pointer on the 64-bit platform), the actual memory size required is 5 * 8 = 40 bytes. The closest 40 bytes of the 67 spans is 48 bytes, so the actual allocated memory capacity is 48 bytes.

The final calculated real capacity is 48 / 8 = 6 , which is consistent with the actual running output of Lao Xu.

Finally, I sincerely hope that this article can be helpful to readers.

Note :

1. At the time of writing this article, the version of go used by the author is: go1.16.6
2. The complete example used in the article: https://github.com/Isites/go-coder/blob/master/slice/main.go