Go1.18 new features: efficient copying, strings, bytes standard library added Clone API

Hello everyone, I am fried fish.

Go1.18 will be released in a few weeks (March). We have updated several new version features before. Today, we bring you a new optimization class, which is related to the strings and bytes standard library.

background

Want to copy faster

In daily programming, the byte []byte is often copied. Need to write the following code:

dup := make([]byte, len(data))
copy(dup, data)

@Ilia Choly thinks this is too much trouble, after all, you have to write it again every time, or encapsulate it into the following function:

// Clone returns a copy of b
func Clone(b []byte) []byte {
  b2 := make([]byte, len(b))
  copy(b2, b)
  return b2
}

To this end, I want to add a shortcut method, but this is obviously untenable. Students who are familiar with grammar will find a ready-made method:

b2 := append([]byte(nil), b...)

The effect can be achieved in one line, and even faster, because the allocated slices are not initialized to zero values.

Copy will share memory

Many Go developers, when writing an application to copy a slice (Slice), will find that the copied slice s1 and the original s0 are associated in memory. The essential reason is the underlying data structure, which will lead to many hidden The problem.

The sample code is as follows:

import (
    "fmt"
    "reflect"
    "unsafe"
)

func main() {
    s0 := "脑子进煎鱼了"
    s1 := s0[:3]
    s0h := (*reflect.StringHeader)(unsafe.Pointer(&s0))
    s1h := (*reflect.StringHeader)(unsafe.Pointer(&s1))

    fmt.Printf("Len is equal: %t\n", s0h.Len == s1h.Len)
    fmt.Printf("Data is equa: %t\n", s0h.Data == s1h.Data)
}

From the above program, do you think that the Len of the variables s0 and s1 are equal? Are Data equal?

The output is as follows:

Len is equal: false
Data is equa: true

At first glance, it seems to be ok. Len is not equal, it is copied by index after all. But Data is actually equal, why is this, is it a bug in Go?

This is actually related to Go's underlying data structures in String and Slice, such as String's runtime representation of StringHeader.

Its underlying structure is as follows:

type StringHeader struct {
    Data uintptr
    Len  int
}

• Data: Points to the specific underlying array.
• Len: represents the length of the string slice.

The key is Data, which is essentially a pointer address to the underlying data, so when copying, it will also be copied over. Causes unnecessary duplication and "dirty" data, causing all kinds of weird bugs.

This problem is the pain point of the new feature.

new features

In the new features of Go1.18, strings and bytes have added a Clone method to solve the two problems mentioned above.

The source code is as follows:

func Clone(s string) string {
    if len(s) == 0 {
        return ""
    }
    b := make([]byte, len(s))
    copy(b, s)
    return *(*string)(unsafe.Pointer(&b))
}

• Copy the original string through the copy function to get a new []byte data.
• Pointer operations are performed through *(*string)(unsafe.Pointer(&b)) to achieve zero memory copy conversion from byte to string .

So far, two problems in the background have been solved in a very clever way. You don't have to keep writing similar code over and over again.

Summarize

For a long time, strings and slices in Go have been controversial because of the underlying array pointing and expansion and contraction mechanisms. At the end, some "pits" that were introduced before the official account are also given. Interested readers can take a look.

The 1.18 new method update this time has a certain function. In learning, you can also clearly know the role and definition of Clone, and you will keep an eye on it.

Do you have similar code in your project?

If you have any questions, welcome feedback and exchange in the comment area. The best relationship between is to achieve each other . Your likes is the biggest driving force for fried fish create, thank you for your support.

The article is updated continuously, you can read it on WeChat search [Brain fried fish], this article GitHub github.com/eddycjy/blog has been included, to learn Go language, you can read Go learning map and route .

Recommended in the past

• Go slicing caused a memory leak and was hacked twice!
• Go slicing this question, noisy all afternoon!

refer to

• bytes, strings: add Clone
• [test: add test that we don't optimize string([]byte(string))](https://github.com/golang/go/issues/25834)
• proposal: provide std lib pkg Clone function to copy a string efficiently without memory sharing