Go Web Service Framework Implementation Detailed Explanation

foreword

This series of articles requires readers to have a certain golang foundation.
go-zero is a web and rpc framework that integrates various engineering practices. The stability of the large concurrent server is guaranteed through the elastic design, which has been fully tested by actual combat.
go-zero includes a minimalist API definition and generation tool goctl, which can generate Go, iOS, Android, Kotlin, Dart, TypeScript, JavaScript code according to the defined api file with one click, and run it directly.

How to interpret a web framework There is no doubt that reading a go web framework is the same as a PHP framework:

1. Configuration Loading: How to load configuration files.
2. Routing: Analyze how the framework executes the corresponding business through the URL.

3. ORM: How ORMs are implemented.
   Among them, 1 and 3 are nothing more than the implementation of loading and parsing configuration files and sql parser, and I will ignore them. Since most of the industry is more performance analysis, I may focus more on the following dimensions:

   • framework design
   • routing algorithm

First of all, we mainly focus on the frame design .

---

Install

To develop a golang program, it is inevitable to install its environment. We choose 1.16.13 as an example here. And using Go Module as a way to manage dependencies is similar to how composer manages dependencies in PHP.
First install the goctl (go control) tool:

goctl is a code generation tool under the go-zero microservice framework. Using goctl can significantly improve development efficiency and allow developers to focus their time on business development. Its functions include:

• API service generation
• rpc service generation
• model code generation
• Template management

 # Go 1.16 及以后版本
GOPROXY=https://goproxy.cn/,direct go install github.com/zeromicro/go-zero/tools/goctl@latest

This command can be used to install the goctl tool to the $GOPATH/bin directory.
Let's take the api service as an example and install it using go mod:

 // 创建项目目录
mkdir zero-demo
cd zero-demo
// 初始化go.mod文件
go mod init zero-demo
// 快捷创建api服务
goctl api new greet
// 安装依赖
go mod tidy
// 复制依赖到vender目录
go mod vendor

At this point, a simple api service is initialized.
Start the service:

 // 默认开启8888端口
go run greet/greet.go -f greet/etc/greet-api.yaml

code analysis

HTTP SERVER

go has its own http package, and most go frameworks are also based on this http package, so let's add or review this knowledge point before looking at go-zero. as follows:
How does GO start a HTTP SERVER

 // main.go
package main

import (
    // 导入net/http包
    "net/http"
)

func main() {
    // ------------------ 使用http包启动一个http服务 方式一 ------------------
    // *http.Request http请求内容实例的指针
    // http.ResponseWriter 写http响应内容的实例
    http.HandleFunc("/v1/demo", func(w http.ResponseWriter, r *http.Request) {
        // 写入响应内容
        w.Write([]byte("Hello World !\n"))
    })
    // 启动一个http服务并监听8888端口 这里第二个参数可以指定handler
    http.ListenAndServe(":8888", nil)
}

// 测试我们的服务
// --------------------
// 启动：go run main.go
// 访问： curl "http://127.0.0.1:8888/v1/demo"
// 响应结果：Hello World !

ListenAndServe is a further encapsulation of http.Server , in addition to the above method, you can also use http.Server to directly start the service, this needs to be set Handler This Handler to implement Server.Handler this interface. When the request comes, this Handler ServeHTTP method will be executed, as follows:

 // main.go
package main

// 导入net/http包
import (
    "net/http"
)

// DemoHandle server handle示例
type DemoHandle struct {
}

// ServeHTTP 匹配到路由后执行的方法
func (DemoHandle) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("Hello World !\n"))
}

func main() {
    // ------------------ 使用http包的Server启动一个http服务 方式二 ------------------
    // 初始化一个http.Server
    server := &http.Server{}
    // 初始化handler并赋值给server.Handler
    server.Handler = DemoHandle{}
    // 绑定地址
    server.Addr = ":8888"

    // 启动一个http服务
    server.ListenAndServe()

}

// 测试我们的服务
// --------------------
// 启动：go run main.go
// 访问： curl "http://127.0.0.1:8888/v1/demo"
// 响应结果：Hello World !

So far we understand the basic server service foundation, let's take a look at how go-zero is used.

Directory Structure

 // 命令行
tree greet

greet
├── etc                                 // 配置
│   └── greet-api.yaml                  // 配置文件
├── greet.api                           // 描述文件用于快速生成代码
├── greet.go                            // 入口文件
└── internal                            // 主要操作文件夹，包括路由、业务等
    ├── config                          // 配置
    │   └── config.go                   // 配置解析映射结构体
    ├── handler                         // 路由
    │   ├── greethandler.go             // 路由对应方法
    │   └── routes.go                   // 路由文件
    ├── logic                           // 业务
    │   └── greetlogic.go
    ├── svc
    │   └── servicecontext.go           // 类似于IOC容器，绑定主要操作依赖
    └── types
        └── types.go                    // 请求及响应结构体

Let's start with the entry file:

 package main

import (
    "flag"
    "fmt"

    "zero-demo/greet/internal/config"
    "zero-demo/greet/internal/handler"
    "zero-demo/greet/internal/svc"

    "github.com/zeromicro/go-zero/core/conf"
    "github.com/zeromicro/go-zero/rest"
)

var configFile = flag.String("f", "etc/greet-api.yaml", "the config file")

func main() {
    // 解析命令
    flag.Parse()
    
    // 读取并映射配置文件到config结构体
    var c config.Config
    conf.MustLoad(*configFile, &c)
    
    // 初始化上下文
    ctx := svc.NewServiceContext(c)
    
    // 初始化服务
    server := rest.MustNewServer(c.RestConf)
    defer server.Stop()

    // 初始化路由及绑定上下文
    handler.RegisterHandlers(server, ctx)

    // 启动服务
    fmt.Printf("Starting server at %s:%d...\n", c.Host, c.Port)
    server.Start()
}

The life cycle of go-zero

The following picture is my output for the entire go-zero framework life cycle:

Visit source image: https://franktrue.oss-cn-shanghai.aliyuncs.com/images/go-zero%27s%20life%20cycle-small.png

Key code analysis

 ⬇️step1
// 获取一个server实例
server := rest.MustNewServer(c.RestConf)
⬇️step2
// 具体的rest.MustNewServer方法
// ----------------------MustNewServer---------------------------
func  MustNewServer(c RestConf, opts ...RunOption) *Server {
    server, err := NewServer(c, opts...)
    if err != nil {
        log.Fatal(err)
    }
    return server
}
⬇️step3
// 创建一个server实例的具体方法
// ---------------------NewServer------------------------------------
func NewServer(c RestConf, opts ...RunOption) (*Server, error) {
    if err := c.SetUp(); err != nil {
        return nil, err
    }

    server := &Server{
        ngin:   newEngine(c),
        router: router.NewRouter(),
    }
    // opts主要是一些对server的自定义操作函数
    opts = append([]RunOption{WithNotFoundHandler(nil)}, opts...)
    for _, opt := range opts {
        opt(server)
    }

    return server, nil
}
⬇️step4
// 上面是一个server实例初始化的关键代码，下面我们分别看下server.ngin和server.router
// -----------------------------engine----------------------------------------
// 创建一个engine
func newEngine(c RestConf) *engine {
    srv := &engine{
        conf: c,
    }
    // Omit the code
    return srv
}

type engine struct {
    conf                 RestConf                       // 配置信息
    routes               []featuredRoutes               // 初始路由组信息
    unauthorizedCallback handler.UnauthorizedCallback   // 认证
    unsignedCallback     handler.UnsignedCallback       // 签名
    middlewares          []Middleware                   // 中间件
    shedder              load.Shedder
    priorityShedder      load.Shedder
    tlsConfig            *tls.Config
}
⬇️step5
// -----------------------------router-------------------------------------------
// 接下来我们看路由注册部分

// 创建一个router
func NewRouter() httpx.Router {
    return &patRouter{
        trees: make(map[string]*search.Tree),
    }
}

// 这里返回了一个实现httpx.Router接口的实例，实现了ServeHttp方法
// ---------------------------Router interface-----------------------------------
type Router interface {
    http.Handler
    Handle(method, path string, handler http.Handler) error
    SetNotFoundHandler(handler http.Handler)
    SetNotAllowedHandler(handler http.Handler)
}
⬇️step6
// 注册请求路由
// 这个方法就是将server.ngin.routes即featuredRoutes映射到路由树trees上
func (pr *patRouter) Handle(method, reqPath string, handler http.Handler) error {
    if !validMethod(method) {
        return ErrInvalidMethod
    }

    if len(reqPath) == 0 || reqPath[0] != '/' {
        return ErrInvalidPath
    }

    cleanPath := path.Clean(reqPath)
    tree, ok := pr.trees[method]
    if ok {
        return tree.Add(cleanPath, handler)
    }

    tree = search.NewTree()
    pr.trees[method] = tree
    return tree.Add(cleanPath, handler)
}
⬇️step7
// 路由树节点
Tree struct {
    root *node
}
node struct {
    item     interface{}
    children [2]map[string]*node    
}

// 上面我们基本看完了server.ngin和server.router的实例化
// ----------------------------------http server------------------------------------
// 接下来我们看下go-zero如何启动http server的
⬇️step8
server.Start()
⬇️step9
func (s *Server) Start() {
    handleError(s.ngin.start(s.router))
}
⬇️step10
func (ng *engine) start(router httpx.Router) error {
    // 绑定路由，将server.ngin.routes即featuredRoutes映射到路由树trees上
    if err := ng.bindRoutes(router); err != nil {
        return err
    }

    if len(ng.conf.CertFile) == 0 && len(ng.conf.KeyFile) == 0 {
        // 无加密证书，则直接通过http启动
        return internal.StartHttp(ng.conf.Host, ng.conf.Port, router)
    }
    // 这里是针对https形式的访问，我们主要看上面的http形式
    return internal.StartHttps(ng.conf.Host, ng.conf.Port, ng.conf.CertFile,
        ng.conf.KeyFile, router, func(srv *http.Server) {
            if ng.tlsConfig != nil {
                srv.TLSConfig = ng.tlsConfig
            }
        })
}
⬇️step11
// 绑定路由
ng.bindRoutes(router)
⬇️step12
// 将server.ngin.routes即featuredRoutes映射到路由树trees上
func (ng *engine) bindRoutes(router httpx.Router) error {
    metrics := ng.createMetrics()

    for _, fr := range ng.routes {
        if err := ng.bindFeaturedRoutes(router, fr, metrics); err != nil {
            return err
        }
    }

    return nil
}
// 映射的同时对每个路由执行中间件操作
func (ng *engine) bindRoute(fr featuredRoutes, router httpx.Router, metrics *stat.Metrics,
    route Route, verifier func(chain alice.Chain) alice.Chain) error {
    // go-zero框架默认中间件
    // ---------------------------------Alice--------------------------------------------
    // Alice提供了一种方便的方法来链接您的HTTP中间件函数和应用程序处理程序。
    //In short, it transforms
    // Middleware1(Middleware2(Middleware3(App)))
    // to
    // alice.New(Middleware1, Middleware2, Middleware3).Then(App)
    // --------------------------------Alice--------------------------------------------
    chain := alice.New(
        handler.TracingHandler(ng.conf.Name, route.Path),
        ng.getLogHandler(),
        handler.PrometheusHandler(route.Path),
        handler.MaxConns(ng.conf.MaxConns),
        handler.BreakerHandler(route.Method, route.Path, metrics),
        handler.SheddingHandler(ng.getShedder(fr.priority), metrics),
        handler.TimeoutHandler(ng.checkedTimeout(fr.timeout)),
        handler.RecoverHandler,
        handler.MetricHandler(metrics),
        handler.MaxBytesHandler(ng.conf.MaxBytes),
        handler.GunzipHandler,
    )
    chain = ng.appendAuthHandler(fr, chain, verifier)
    // 自定义的全局中间件
    for _, middleware := range ng.middlewares {
        chain = chain.Append(convertMiddleware(middleware))
    }
    handle := chain.ThenFunc(route.Handler)

    return router.Handle(route.Method, route.Path, handle)
}
⬇️step13
internal.StartHttp(ng.conf.Host, ng.conf.Port, router)
⬇️step14
func StartHttp(host string, port int, handler http.Handler, opts ...StartOption) error {
    return start(host, port, handler, func(srv *http.Server) error {
        return srv.ListenAndServe()
    }, opts...)
}
⬇️step15
func start(host string, port int, handler http.Handler, run func(srv *http.Server) error,
    opts ...StartOption) (err error) {
    server := &http.Server{
        Addr:    fmt.Sprintf("%s:%d", host, port),
        Handler: handler,
    }
    for _, opt := range opts {
        opt(server)
    }

    waitForCalled := proc.AddWrapUpListener(func() {
        if e := server.Shutdown(context.Background()); err != nil {
            logx.Error(e)
        }
    })
    defer func() {
        if err == http.ErrServerClosed {
            waitForCalled()
        }
    }()
    // run即上一步中的srv.ListenAndServe()操作，因为server实现了ServeHttp方法
    // 最终走到了http包的Server启动一个http服务（上文中http原理中的方式二）
    return run(server)
}

Epilogue

Finally, let's briefly review the above process. From the figure below, it is relatively easy to understand.

---

refer to

https://www.bilibili.com/video/BV1d34y1t7P9 Code explanation of Mikael's api service

project address

https://github.com/zeromicro/go-zero

Welcome go-zero and star support us!

WeChat exchange group

Follow the official account of " Microservice Practice " and click on the exchange group to get the QR code of the community group.