go-zero distributed transaction practice

background

With the rapid development of business and increasing business complexity, microservices are one of the best solutions. While decoupling services, reducing complexity, and increasing maintainability, it also brings some new problems.

When we need to ensure data consistency across services, the original database transaction is not enough, and it is impossible to put multiple operations across databases and services in one transaction. There are many such application scenarios, and we can list many:

• In cross-bank transfer scenarios, the data is not in the same database, but it is necessary to ensure that the balance deduction and balance increase either succeed or fail at the same time
• After the article is published, update statistics such as the total number of articles. Publish articles and update statistics are usually in different microservices
• Order system after microservice
• Traveling requires several tickets to be booked in the third-party system at the same time

In the face of scenarios where these local transactions cannot be resolved, we need a distributed transaction solution to ensure the consistency of cross-service and cross-database update data.

Go-zero and dtm have joined forces to launch a minimalist solution that seamlessly connects to dtm in go-zero, making the use of distributed transactions easier than ever.

Run an example

Let's look at a runnable example, and then look at how to develop a complete distributed transaction by ourselves

With etcd as the registration service center below, you can run a go-zero example according to the following steps:

• Configure dtm

  MicroService:
    Driver: 'dtm-driver-gozero' # 配置dtm使用go-zero的微服务协议
    Target: 'etcd://localhost:2379/dtmservice' # 把dtm注册到etcd的这个地址
    EndPoint: 'localhost:36790' # dtm的本地地址

• Start etcd

  # 前提：已安装etcd
  etcd

• Start dtm

  # 前提：已配置好dtm的数据库链接
  go run app/main.go dev

• Run a go-zero service

  git clone github.com/yedf/dtmdriver-clients && cd dtmdriver-clients
  cd gozero/trans && go run trans.go

• Use go-zero to initiate a dtm transaction

  # 在dtmdriver-clients的目录下
  cd gozero/app && go run main.go

When you see in the trans log

2021/12/03 15:44:05 transfer out 30 cents from 1
2021/12/03 15:44:05 transfer in 30 cents to 2
2021/12/03 15:44:05 transfer out 30 cents from 1
2021/12/03 15:44:05 transfer out 30 cents from 1

That is the transaction is completed normally

Development access

Refer to the code of yedf/dtmdriver-clients

// 下面这行导入gozero的dtm驱动
import _ "github.com/yedf/dtmdriver-gozero"

// 使用dtm的客户端dtmgrpc之前，需要执行下面这行调用，告知dtmgrpc使用gozero的驱动来如何处理gozero的url
err := dtmdriver.Use("dtm-driver-gozero")
// check err

// dtm已经通过前面的配置，注册到下面这个地址，因此在dtmgrpc中使用该地址
var dtmServer = "etcd://localhost:2379/dtmservice"

// 下面从配置文件中Load配置，然后通过BuildTarget获得业务服务的地址
var c zrpc.RpcClientConf
conf.MustLoad(*configFile, &c)
busiServer, err := c.BuildTarget()

  // 使用dtmgrpc生成一个消息型分布式事务并提交
    gid := dtmgrpc.MustGenGid(dtmServer)
    msg := dtmgrpc.NewMsgGrpc(dtmServer, gid).
    // 事务的第一步为调用trans.TransSvcClient.TransOut
    // 可以从trans.pb.go中找到上述方法对应的Method名称为"/trans.TransSvc/TransOut"
    // dtm需要从dtm服务器调用该方法，所以不走强类型，而是走动态的url: busiServer+"/trans.TransSvc/TransOut"
        Add(busiServer+"/trans.TransSvc/TransOut", &busi.BusiReq{Amount: 30, UserId: 1}).
        Add(busiServer+"/trans.TransSvc/TransIn", &busi.BusiReq{Amount: 30, UserId: 2})
    err := msg.Submit()

The entire development access process is very small, the previous comments are already very clear, so I won’t repeat it

Precautions

In the process of development and access, when looking for the method path of grpc access in the *.pb.go file, you must find the path of invoke

Deep understanding of dynamic calls

When go-zero uses dtm's distributed transactions, many calls are initiated from the dtm server, such as TCC's Confirm/Cancel, and all SAGA/MSG calls.

dtm does not need to know the strong types of related business APIs that make up distributed transactions. It calls these APIs dynamically.

The call of grpc can be analogous to HTTP POST, where:

• The target generated by c.BuildTarget() is similar to the Host in the URL
• "/trans.TransSvc/TransOut" is equivalent to Path in URL
• &busi.BusiReq{Amount: 30, UserId: 1} is equivalent to Body in Post
• pb.Response is equivalent to the response of the HTTP request

Through the following part of the code, dtm has obtained the complete information and can initiate a complete call

Add(busiServer+"/trans.TransSvc/TransOut", &busi.BusiReq{Amount: 30, UserId: 1})

More complete example

Enthusiastic community classmate Mikael helped write a richer example, combined with practical applications and sub-transaction barriers, a complete demonstration of a distributed transaction actually running online. Interested students can refer to:

https://github.com/Mikaelemmmm/gozerodtm

Access in other ways

Go-zero’s microservices also have other methods other than etcd. We will explain their access methods in turn

Direct connection

For the direct connection method, you only need to set Target to an empty string based on the etcd configuration of dtm above.

In the case of direct connection, there is no need to register dtm to the registration center

K8S

For K8S, you only need to set Target to an empty string based on the etcd configuration of dtm above.

In K8S, the registration of the service to K8S is done by deployment.yaml, inside the application, no registration is required

Live sharing preview

The author of go-zero and I (the author of dtm) will jointly do a live sharing of "Go-zero Distributed Transaction Practice" on talkgo at 21:00 on December 22, which will bring more changes. In-depth discussion. Everyone is welcome to attend.

The live broadcast address is: https://live.bilibili.com/11171965

summary

This time, the cooperation between go-zero and dtm has created the first microservice solution that natively supports distributed transactions in the go ecosystem, which is of great significance.

• go-zero project address: https://github.com/zeromicro/go-zero
• dtm project address: https://github.com/yedf/dtm

Welcome everyone to use our go-zero and dtm , use our native "distributed transaction microservice solution", and star support us