The Go program is too big, can we have lazy initialization?

Hello everyone, I am fried fish.

In the continuous development of the company, most of them are large monomers at the beginning, and the transformation is slow. A warehouse will be used for decades, and the scale of the warehouse is basically a process of continuous increase.

One of the effects is that the packaged application will become larger and larger, and I don't know where it is used... The proposal to be discussed today, " proposal: language: lazy init imports to possibly import without side effects ", is the same as this related.

proposal

background

Let's look at a very simple Go code and study it. The following code:

 package main

import _ "crypto/x509"

func main() {}

This Go program is only 3 lines of code and doesn't look like anything. Is that actually the case?

We can execute the following command to see the initialization process:

 $ go build --ldflags=--dumpdep main.go 2>&1 | grep inittask

Output result:

 runtime.main -> runtime..inittask
runtime.main -> main..inittask
main..inittask -> crypto/x509..inittask
crypto/x509..inittask -> bytes..inittask
crypto/x509..inittask -> crypto/sha256..inittask
crypto/x509..inittask -> encoding/pem..inittask
crypto/x509..inittask -> errors..inittask
crypto/x509..inittask -> sync..inittask
crypto/x509..inittask -> crypto/aes..inittask
crypto/x509..inittask -> crypto/cipher..inittask
crypto/x509..inittask -> crypto/des..inittask
...
context..inittask -> context.init.0
vendor/golang.org/x/net/dns/dnsmessage..inittask -> vendor/golang.org/x/net/dns/dnsmessage.init
vendor/golang.org/x/net/route..inittask -> vendor/golang.org/x/net/route.init
vendor/golang.org/x/net/route..inittask -> vendor/golang.org/x/net/route.init.0
...

This program actually initializes a lot of software packages (standard library, third-party packages, etc.). Makes the size of the package from the standard 1.3 MB to 2.3 MB.

At a certain scale, the impact is thought to be very expensive. Because you can see that the Go program with only 3 lines doesn't do anything substantial.

Programs that are sensitive to startup performance will be more uncomfortable, and ordinary programs will also enter a vicious circle over time, and the startup will be slower than normal.

Program

For the solution, let's look at it together with another proposal " proposal: spec: Go 2: allow manual control over imported package initialization ".

The core idea is to introduce lazy initialization (lazy init), also known as lazy loading in the industry. That is to say, it is really imported when necessary, and the initialization is not completed when the package is imported.

In the direction of optimization: mainly add lazy initialization declarations after importing the package path, such as the go:lazyinit or go:deferred annotations that will be mentioned below. Then wait until the program is actually used before it is officially initialized.

1. Example of go:lazyinit:

 package main

import (
      "crypto/x509" // go:lazyinit
      "fmt"
)

func main() {...}

2. Example of go:deferred:

 package main

import (
    _ "github.com/eddycjy/core" // go:deferred
    _ "github.com/eddycjy/util" // go:deferred
)

func main() {
    if os.Args[1] != "util" {
        // 现在要使用这个包，开始初始化
        core, err := runtime.InitDeferredImport("github.com/some/module/core")
        ...
    }
    ...
}

In this way, the startup performance can be greatly improved.

discuss

In fact, in most community discussions, there is a love-hate relationship with this proposal. Because it seems to have a reasonable appeal, but it seems to be completely wrong when you think about it.

The background and solution of this proposal is to treat the symptoms rather than the root causes. Because the root cause is this: many libraries abuse the init function and initialize a lot of unnecessary things.

The Go core development team believes that it is up to the library authors to fix these libraries, rather than letting Go "fix" the problems. Supporting lazy initialization would also give authors of these low-quality libraries an excuse to continue doing so.

deja vu

While writing this article, I was reminded of Go's dependency management (Go modules), which has a design based on the Semantic Versioning specification.

As shown below

The version format is "major version number. minor version number. revision number", and the increment rules for version numbers are as follows:

• Major version number: When you make an incompatible API change.
• Minor version number: When you make a functional addition for backward compatibility.
• Revision number: When you make corrections for backward compatibility issues.

The original intention of Go modules is that all software libraries comply with this specification, so there will be a minimum version selection logic inside.

That is, a module often depends on many other modules, and different modules are likely to depend on different versions of the same module when they are dependent. Go will sort out the version list and finally get a build list.

As shown below:

You will find that the final build of the dependency version is likely to be inconsistent with the expected, causing many business problems. The most classic is the multi-version compatibility problem of grpc-go, protoc-go, etcd, which makes many people suffer.

The design of the Go team in this area is more ideal, and Cao Da also classified it as one of the seven deadly sins of Go modules. The init function of the software package initializes a bunch of problems, which is also somewhat familiar.

Summarize

The solution (proposal) to this problem is still under discussion. Obviously, the Go team prefers that the authors of the software library can restrain their own code and not initialize randomly.

What is the way to introduce lazy initialization, what do you think? Welcome to leave a message and discuss in the comment area.

The article is continuously updated, you can read it on WeChat by searching [Brain Fried Fish]. This article has been included in GitHub github.com/eddycjy/blog . To learn Go language, you can see the Go learning map and route . Welcome to Star to urge you to update.

Go Book Series

• Introduction to the Go language series: a preliminary exploration of the actual combat of the Go project
• Go Programming Journey: Deep Dive into Go Projects
• Go Language Design Philosophy: Understanding Go Why and Design Thinking
• Go Language Advanced Tour: Go deeper into the Go source code

Recommended reading

• Go1.19 Those things: How much do you know about new features such as domestic chips and memory models?
• It's crazy that the Go program says nil is not nil...