New in Go1.18: Introducing a new netip network library

Hello everyone, I am fried fish.

When I wrote this article, it was the first day of the new year, and I originally wanted to say that Go1.18 would be released this month. But, good guy, Go1.18 beta2 has been released, and the official told the community that Go1.18 will be delayed until March, gugugu...

As shown below:

So you still have to continue to learn new features. Today, Jianyu will combine with Brad Fitzpatrick's " netaddr.IP: a new IP address type for Go " to show you the reason for Go1.18's new network library net/netip.

background

Big boss resigns

Brad Fitzpatrick, who was in the original Go development team, worked in the Go team from 2010 to 2020, and will change companies in 2021.

The following tweet:

The reason for leaving: I've been doing the same thing for too long, I'm a little bored, I don't want to be stuck in a comfortable predicament.

Now it seems that I have switched to Tailscale to do WireGuard related work, and I have to deal with network libraries frequently.

Demand is born

Tailscale written by the big brother company is essentially a network application. To deal with the network and written in Go, it will involve the standard library net :

• Use net.IP on a single IP type.
• net.IPNet is used on the network representation.

Sample code:

import (
    "fmt"
    "net"
)

func main() {
    fmt.Println(net.IPv4(8, 8, 8, 8))
}

Output result:

8.8.8.8

When Brad Fitzpatrick actually wrote and used it, he found that there are many problems with the type of the net standard library, which is very difficult to use.

what's the problem now

Brad Fitzpatrick listed the problems of the standard library net.IP directly in the article, with full arguments.

A total of 7 major questions:

1. It is mutable.
   The underlying type of net.IP is []byte , which means that anything you pass to it may change it.
2. It is not comparable. Because slices in Go are not comparable, this means that net.IP does not support the comparison of Go's == operator and cannot be used as a map key.
3. It has two IP address types, and it can be annoying to choose between net.IP or net.IPAddr .
4. It's huge. Go's net.IP consists of 2 parts, a 24-byte slice header and a 4/6-byte IP address. If it is net.IPAddr it will also contain the Zone field.
5. It will allocate memory on the heap. Go's net package is allocated all over the place, putting more work on the GC.
6. It is not parseable. When parsing IP from string form, Go's IP type cannot distinguish between IPv4-mapped IPv6 addresses and IPv4 addresses.
7. It is a transparent type, net.IP is defined as: type IP []byte , is part of its public API and cannot be changed.

Brad also mentioned that some of the designs were early in the year, when they were inexperienced or not well thought out.

It is now limited by the Go1 compatibility promise and cannot be changed (a double-edged sword of compatibility guarantees?).

This is a real version of "Eating your own dog food", so at Tailscale he rebuilt another wheel inetaf/netaddr and wanted to contribute it to the standard library.

the future you want

The comparison table is as follows:

characteristic	Old scheme net.IP	new plan
Changeless	❌, slice	✅
comparable	❌, slice	✅
Small footprint	❌, 28~56 bytes	✅, fixed 24 bytes
not allocated on the heap	❌	✅
Supports IPv4 and IPv6	✅	✅
Differentiate between IPv4 and IPv6	❌	✅
IPv6 zone support	❌	✅
opaque type	❌	✅
Interoperate with the standard library	✅	🤷, need to adapt method

What I want is actually a demand from Brad's actual business, which is to support the 7 points mentioned above.

solution

current progress

The result of the realization, that is, the new solution is made, he is the library inetaf/netaddr (of course, it is not ruled out that the result is reversed). And initiate issues and proposals in Go issues.

Russ Cox initiated the discussion of the new proposal " proposal: net/netaddr: add new IP address type, netaddr package (discussion) ")", and was accepted into the new features of Go1.18.

rebuild process

Every consideration of the new net/netip library is explained in detail by Brad in the article.

Due to space limitations, we will share two of them. Interested partners can read the analysis part of the original text.

interface type combination

In terms of comparability, Go's interface actually supports comparison, that is, it can be used as the key of the map to compare the == operator.

The first version of the following scheme is implemented, and a new netaddr.IP type is designed:

type IP struct {
  ipImpl
}

type ipImpl interface {
  is4() bool
  is6() bool
  String() string
}

type v4Addr [4]byte
type v6Addr [16]byte
type v6AddrZone struct {
  v6Addr
  zone string
}

The above code adds the ipImpl interface to the IP structure, which can not only support comparison, but also not exposed to the outside world (opaque type), and can support IPv6.

The new problem is that although it is smaller than the native net, it still does not achieve the goal, and it still has the disadvantage of allocating on the heap.

Allocation-free 24 bytes

If you continue to use the interface, you cannot solve the underlying goal (Brad's goal is 24 bytes).

Because the interface occupies 16 bytes, the remaining 8 bytes can be used, and the following things should be placed:

• Address family (v4, v6, or neither, eg: zero for IP), at least 2 bits required.
• IPv6 zone information.

It is also necessary to compare, obviously the interface cannot be realized, because the address + zone information counts the number of bytes, and the display is not enough.

There is no way to do it formally and explicitly, Brad thought of using the packaging method:

type IP struct {
   addr          [16]byte
   zoneAndFamily uint64
}

But doing this means that the number of digits in the zoneAndFamily field needs to be calculated, and then the corresponding value is pushed in, but it may not be too much trouble.

In the end Brad came up with a way to use pointers:

type IP struct {
    addr          [16]byte
    zoneAndFamily *T
}

Then define three corresponding sentinel values to apply:

var (
     z0    *intern.Value        // 表示零值。
     z4    = new(intern.Value)  // 表示 IPv4 的哨位值
     z6noz = new(intern.Value)  // 表示 IPv6 的哨位值（没有 zone）。
)

This fixes the IP type at 24 bytes.

Summarize

This network address library is generally used less. But Brad Fitzpatrick put a lot of energy and research into it to get to the end.

In addition to the functions of the library, there are many technical optimization points worthy of our study and reference. If you are interested in in-depth optimization, you can read: https://tailscale.com/blog/netaddr-new-ip-type-for-go/

The new net/netip library introduced in this article will appear as a new feature in Go1.18, and everyone is welcome to learn and communicate together :)

If you have any questions, welcome to feedback and exchange in the comment area. The best relationship between is to each other. Your likes is the biggest driving force for fried fish create, thank you for your support.

The article is continuously updated, you can read it on WeChat search [Brain into Fried Fish], this article GitHub github.com/eddycjy/blog has been included, learn Go language, you can see Go learning map and route