go-zero code generator helps you develop efficiently

Protocol Buffers is an encoding standard introduced by Google, which is superior to JSON in transmission efficiency and encoding and decoding performance. But the price is that you need to rely on the Intermediate Description Language (IDL) to define the structure of data and services (via the *.proto file), and a complete set of toolchains (protoc and its plugins) are required to generate the corresponding sequences code for deserialization and deserialization. In addition to the tools and plug-ins officially provided by Google (such as protoc-gen-go for generating go code), developers can also develop or customize their own plug-ins to generate code or documents according to the definition of proto files according to business needs.

And goctl rpc the purpose of code generation tool development:

1. proto template generation
2. rpc server code generation → get go-zero zrpc
3. Internally wraps the generation of gRPC pb code
4. Like the http server, it provides some built-in control middleware go-zero

We can notice the third point, basically without using the codegen tool, the developer needs to execute the protoc + protoc-gen-go plugin to generate the corresponding .pb.go file. The whole process is rather cumbersome.

The above is the background of goctl rpc . This article first describes the goctl generation process from the perspective of overall generation, and then analyzes some key parts, so that developers can develop codegen tools that fit their business systems.

the whole frame

 // 推荐使用 v3 版本。现在流行的 gRPC 框架也是使用 v3 版本。
syntax = "proto3";

// 每个 proto 文件需要定义自己的包名，类似 c++ 的名称空间。
package hello;

// 数据结构通过 message 定义
message Echo {
  // 每个 message 可以有多个 field。
  // 每个 field 需要指定类型、字段名和编号。
  // Protocol Buffers 在内部使用编号区分字段，一旦指定就不能更改。
  string msg = 1;
}

// 服务使用 servcie 定义
service Demo {
  // 每个 service 可以定义多个 rpc
  // 每个 rpc 需要指定接口名、传入消息和返回消息三部分。
  rpc Echo(Echo) returns (Echo);
}

The so-called code generation is actually to parse out each part of the proto file (IDL), and then render the template corresponding to each part to generate the corresponding code.

And in the generation process, we can also use plug-ins or customize our own plug-ins.

Let's look at the entrance first:

 {
        Name:        "protoc",
        Usage:       "generate grpc code",
        UsageText:   "example: goctl rpc protoc xx.proto --go_out=./pb --go-grpc_out=./pb --zrpc_out=.",
        Description: "for details, see https://go-zero.dev/cn/goctl-rpc.html",
        Action:      rpc.ZRPC,
        Flags:       []cli.Flag{
            ...
        },
}

Enter from goctl.go (basically, the command entries under goctl can be found in this file):

 // ZRPC generates grpc code directly by protoc and generates
// zrpc code by goctl.
func ZRPC(c *cli.Context) error {
        ...

        grpcOutList := c.StringSlice("go-grpc_out")
        goOutList := c.StringSlice("go_out")
        zrpcOut := c.String("zrpc_out")
        style := c.String("style")
        home := c.String("home")
        remote := c.String("remote")
        branch := c.String("branch")
        ...
        goOut := goOutList[len(goOutList)-1]
        grpcOut := grpcOutList[len(grpcOutList)-1]
        ...
    
        var ctx generator.ZRpcContext
        ...
        // 将args中的值逐个赋值给 ZRpcContext，作为env context注入 generator
        g, err := generator.NewDefaultRPCGenerator(style, generator.WithZRpcContext(&ctx))
        if err != nil {
            return err
        }

        return g.Generate(source, zrpcOut, nil)
}

g.Generate(source, zrpcOut, nil) → The core function generated by goctl rpc is responsible for the entire life cycle:

1. parse → proto parse
2. template filling → proto item into template
3. file generation → touch generate file

generator

 func (g *RPCGenerator) Generate(src, target string, protoImportPath []string, goOptions ...string) error {
        ...
        // proto parser
        p := parser.NewDefaultProtoParser()
        proto, err := p.Parse(src)
        
        dirCtx, err := mkdir(projectCtx, proto, g.cfg, g.ctx)
        
        // generate Go code
        err = g.g.GenEtc(dirCtx, proto, g.cfg)
        err = g.g.GenPb(dirCtx, protoImportPath, proto, g.cfg, g.ctx, goOptions...)
        err = g.g.GenConfig(dirCtx, proto, g.cfg)
        err = g.g.GenSvc(dirCtx, proto, g.cfg)
        err = g.g.GenLogic(dirCtx, proto, g.cfg)
        err = g.g.GenServer(dirCtx, proto, g.cfg)
        err = g.g.GenMain(dirCtx, proto, g.cfg)
        err = g.g.GenCall(dirCtx, proto, g.cfg)
        ...
}

The above figure shows the code generation process of Generate() .

---

Here are some of the processes GenPb() are explained in advance. Why say this? goctl is a tool system separated from protoc, including the protoc plug-in mechanism, so to generate .pb.go files, how are they coupled?

First, check whether there is a built-in xxx plug-in. If there is no built-in xxx plug-in, it will continue to check whether there is an executable program named protoc-gen-xxx in the current system, and finally generate code through the queried plug-in.

go-zero is not right protoc additionally write a plug-in to help generate code. So by default, the go code generated by protoc-gen-xxx is used.

 func (g *DefaultGenerator) GenPb(ctx DirContext, 
        protoImportPath []string, 
        proto parser.Proto, 
        _ *conf.Config, 
        c *ZRpcContext, 
        goOptions ...string) error {
    ...
    // protoc 命令string
    cw := new(bytes.Buffer)
    ...
    // cw.WriteString("protoc ")
    // cw.WriteString(some command shell)
    command := cw.String()
    g.log.Debug(command)
    _, err := execx.Run(command, "")
    if err != nil {
        if strings.Contains(err.Error(), googleProtocGenGoErr) {
            return errors.New(`unsupported plugin protoc-gen-go which installed from the following source:
google.golang.org/protobuf/cmd/protoc-gen-go, 
github.com/protocolbuffers/protobuf-go/cmd/protoc-gen-go;

Please replace it by the following command, we recommend to use version before v1.3.5:
go get -u github.com/golang/protobuf/protoc-gen-go`)
        }

        return err
    }
    return nil
}

One sentence description GenPb() :

According to the structure and path parsed by the previous proto parse protoc compile and run the command, and then execx.Run(command, "") execute this command directly.

So if developers need to add their own plug-ins, they can modify them by themselves cw.WriteString(some command shell) and write their own execution command logic.

Summarize

That's all for this article. This article analyzes the entire generation process from the entry of goctl rpc to generate rpc code. It specifically mentions the generation of .pb.go files. Developers can cut into goctl rpc from this code part and add their own proto plugins. Of course, there are other parts that will continue to be analyzed in subsequent articles.

The purpose of this series of articles: By the way, I will take you to transform a rpc codegen tool of your own.

project address

https://github.com/zeromicro/go-zero

Welcome go-zero and star support us!

WeChat exchange group

Follow the official account of " Microservice Practice " and click on the exchange group to get the QR code of the community group.