Go language: SliceHeader, how does slice process data efficiently?

Array

In Go language, the array type includes two parts: size of the array, and the internal element type of the array .

a1 := [1]string("微客鸟窝")
a2 := [2]string("微客鸟窝")

In the example, the type of variable a1 is [1] string, and the type of variable a2 is [2] string, because they are not of the same type because of their different sizes.

Array limitations

• After the array is declared, its size and the type of internal elements can no longer be changed
• Because in the Go language, the parameter transfer between functions is value transfer . When the array is used as a parameter, it will be copied. If it is very large, will cause a lot of memory waste

It is precisely because of these limitations of arrays that Go has designed slices again!

slice

The underlying data of slice is stored in an array, which can be said to be an improved version of an array. Slice is an abstraction and encapsulation of an array. It can dynamically add elements and automatically expand when the capacity is insufficient.

## Dynamic expansion
Through the built-in append method, you can append any number of elements to the slice:

func main() {
  s := []string{"微客鸟窝","无尘"}
  s = append(s,"wucs")
  fmt.Println(s) //[微客鸟窝 无尘 wucs]
}

When the append method appends elements, if the capacity of the slice is not enough, it will automatically expand:

func main() {
   s := []string{"微客鸟窝","无尘"}
   fmt.Println("切片长度：",len(s),"；切片容量：",cap(s))
   s = append(s,"wucs")
   fmt.Println("切片长度：",len(s),"；切片容量：",cap(s))
   fmt.Println(s) //
}

operation result:

切片长度： 2 ；切片容量： 2
切片长度： 3 ；切片容量： 4
[微客鸟窝 无尘 wucs]

Through the running results, we found that before calling append, the capacity is 2 and after calling the capacity is 4, indicating that the capacity is automatically expanded.
The principle of expansion is to create a new bottom array, copy the elements in the original slice to the new array, and then return a slice pointing to the new array.

Slice structure

A slice is actually a structure, and its definition is as follows:

type SliceHeader struct {
   Data uintptr
   Len  int
   Cap  int
}

• Data is used to point to the array where the slice elements are stored.
• Len represents the length of the slice.

• Cap represents the capacity of the slice.
  Through these three fields, an array can be abstracted into a slice, so the underlying Data corresponding to different slices may point to the same array.
  Example:

  func main() {
    a1 := [2]string{"微客鸟窝","无尘"}
    s1 := a1[0:1]
    s2 := a1[:]
    fmt.Println((*reflect.SliceHeader)(unsafe.Pointer(&s1)).Data)
    fmt.Println((*reflect.SliceHeader)(unsafe.Pointer(&s2)).Data)
  }

  operation result:

  824634892120
  824634892120

  We found that the Data values printed out of s1 and s2 are the same, indicating that the two slices share an array. Therefore, when operating on slices, the same array is used instead of copying the original elements, which reduces memory usage and improves efficiency.
  Although multiple slices sharing the same underlying array can reduce memory usage, if one slice modifies internal elements, other slices will also be affected, so be careful when passing slices as parameters and try not to modify elements in far slices.
  The essence of slice is SliceHeader, and because the parameter of the function is passed by value, what is passed is a copy of SliceHeader instead of a copy of the underlying array, which can greatly reduce memory usage.
  To get the values of the three fields of the slice array result, in addition to using SliceHeader, you can also customize a structure. Only the package subfield is the same as SliceHeader:

  func main() {
    s := []string{"微客鸟窝","无尘","wucs"}
    s1 := (*any)(unsafe.Pointer(&s))
    fmt.Println(s1.Data,s1.Len,s1.Cap) //824634892104 3 3
  }
  type any struct {
    Data uintptr
    Len int
    Cap int
  }

Efficient

For the collection types in the Go language: array, slice, map, the value and assignment operations of array and slice are more efficient than map, because they are continuous memory operations, and the address of element storage can be quickly found by index . In function parameter passing, slices are more efficient than arrays, because slices are used as parameters and will not copy all the elements. It just copies the three fields of SliceHeader, and the same underlying array is shared.
Example:

func main() {
    a := [2]string{"微客鸟窝", "无尘"}
    fmt.Printf("函数main数组指针：%p\n", &a)

    arrayData(a)
    s := a[0:1]
    fmt.Println((*reflect.SliceHeader)(unsafe.Pointer(&s)).Data)
    sliceData(s)
}
func arrayData(a [2]string) {
    fmt.Printf("函数arrayData数组指针：%p\n", &a)
}
func sliceData(s []string) {
    fmt.Println("函数sliceData数组指针：", (*reflect.SliceHeader)(unsafe.Pointer(&s)).Data)
}

operation result:

函数main数组指针：0xc0000503c0
函数arrayData数组指针：0xc000050400
824634049472
函数sliceData数组指针： 824634049472

it can be discovered:

• The pointer of the same array passed to the arrayData function has changed, indicating that the array was copied when passing parameters, and a new array was created.
• The slice is passed as a parameter to the sliceData function, and the pointer does not change, because the underlying Data of the slice is the same, and the slice shares a single underlying array, and the underlying array is not copied.

Convert between string and []byte

String underlying structure StringHeader:

// StringHeader is the runtime representation of a string.
type StringHeader struct {
   Data uintptr
   Len  int
}

StringHeader, like SliceHeader, represents the real structure of the string when the program is running. It can be seen that the field only has one Cap attribute less than the slice.
[]byte(s) and string(b) forced conversion:

func main() {
   s := "微客鸟窝"
   fmt.Printf("s的内存地址：%d\n", (*reflect.StringHeader)(unsafe.Pointer(&s)).Data)
   b := []byte(s)
   fmt.Printf("b的内存地址：%d\n", (*reflect.SliceHeader)(unsafe.Pointer(&b)).Data)
   c := string(b)
   fmt.Printf("c的内存地址：%d\n", (*reflect.StringHeader)(unsafe.Pointer(&c)).Data)
}

operation result:

s的内存地址：8125426
b的内存地址：824634892016
c的内存地址：824634891984

Through the above example, it is found that the printed memory addresses are different. It can be seen that the forced conversion of []byte(s) and string(b) will copy a string again. If the string is very large, this re-copying method will greatly affect the performance.

optimization

[]byte to string is equivalent to SliceHeader to StringHeader through unsafe.Pointer, which is []byte to string.
Zero copy example:

func main() {
    s := "微客鸟窝"
    fmt.Printf("s的内存地址：%d\n", (*reflect.StringHeader)(unsafe.Pointer(&s)).Data)
    b := []byte(s)
    fmt.Printf("b的内存地址：%d\n", (*reflect.SliceHeader)(unsafe.Pointer(&b)).Data)
    //c1 :=string(b)
    c2 := *(*string)(unsafe.Pointer(&b))
    fmt.Printf("c2的内存地址：%d\n", (*reflect.StringHeader)(unsafe.Pointer(&c2)).Data)
}

operation result:

s的内存地址：1899506
b的内存地址：824634597104
c2的内存地址：824634597104

In the example, the contents of c1 and c2 are the same, the difference is that c2 does not apply for new memory (zero copy), c2 and variable b use the same memory, because their underlying Data field values are the same, which saves Memory also achieves the purpose of converting []byte to string.
SliceHeader has three fields: Data, Len, and Cap. StringHeader has two fields: Data and Len. Therefore, when SliceHeader is converted to StringHeader through unsafe.Pointer, there is no problem, but the reverse is not possible, because StringHeader lacks SliceHeader. The required Cap field, we need to fill in a default value by ourselves:

func main() {
    s := "微客鸟窝"
    fmt.Printf("s的内存地址：%d\n", (*reflect.StringHeader)(unsafe.Pointer(&s)).Data)
    b := []byte(s)
    fmt.Printf("b的内存地址：%d\n", (*reflect.SliceHeader)(unsafe.Pointer(&b)).Data)
    sh := (*reflect.SliceHeader)(unsafe.Pointer(&s))
    sh.Cap = sh.Len
    b1 := *(*[]byte)(unsafe.Pointer(sh))
    fmt.Printf("b1的内存地址：%d\n", (*reflect.StringHeader)(unsafe.Pointer(&b1)).Data)
}

operation result:

s的内存地址：1309682
b的内存地址：824634892008
b1的内存地址：1309682

1. The contents of b1 and b are the same. The difference is that b1 does not apply for new memory, but uses the same memory as the variable s. Because their underlying Data fields are the same, memory is also saved.
2. After converting string to []byte through unsafe.Pointer, []byte cannot be modified. For example, b1[0]=10 cannot be performed. An exception will be reported and the program will crash. Because the string memory is read-only in the Go language.