Distributed transactions across Mysql, Redis, Mongo

Mysql, Redis, Mongo are all very popular storage, and each has its own advantages. In practical applications, multiple storages are often used at the same time, and there is also a need to ensure data consistency in multiple storages, such as ensuring that the inventory in the database is consistent with the inventory in Redis.

Based on the distributed transaction framework https://github.com/dtm-labs/dtm , this article gives a runnable distributed transaction instance across multiple storage engines such as Mysql, Redis, and Mongo, hoping to help you solve this problem The problem.

This ability to flexibly combine multiple storage engines to form a distributed transaction is also pioneered by dtm, and no other distributed transaction framework has seen such a capability.

problem scenario

Let's look at the problem scenario first. Suppose now that the user participates in an event, recharges their balance and calls, and the event will give away mall points. The balance is stored in Mysql, the call charge is stored in Redis, and the mall points are stored in Mongo. Due to the limited time of the event, it may fail to participate in the event, so it needs to support rollback.

For the above problem scenarios, DTM's Saga transaction can be used. Let's explain the solution in detail.

Prepare data

The first is to prepare the data. In order to facilitate users to quickly get started with related examples, we have prepared the relevant data. The address is at en.dtm.pub, which includes Mysql, Redis, and Mongo. The specific connection username and password can be found at https: //github.com/dtm-labs/dtm-examples .

If you want to prepare the relevant data environment locally, you can start Mysql, Redis, Mongo through https://github.com/dtm-labs/dtm/blob/main/helper/compose.store.yml , and then use https://github.com/dtm-labs/dtm/blob/main/helper/compose.store.yml The script below ://github.com/dtm-labs/dtm/tree/main/sqls prepares the data of this example, where busi.* is the business data, barrier.* is used by DTM Auxiliary table

Write business code

Let's first look at the most familiar Mysql business code

 func SagaAdjustBalance(db dtmcli.DB, uid int, amount int) error {
    _, err := dtmimp.DBExec(db, "update dtm_busi.user_account set balance = balance + ? where user_id = ?", amount, uid)
    return err
}

This code is mainly to adjust the user balance in the database

For the Saga transaction mode, when we roll back, we need to reversely adjust the balance. For this part of the processing, we can still call the above SagaAdjustBalance , only need to pass in the negative amount.

For Redis and Mongo, the processing of business code is similar, and it is only necessary to increase or decrease the corresponding balance.

How to do idempotent

For the Saga transaction mode, when our sub-transaction service has a temporary failure, the failure will be retried. This failure may occur before the sub-transaction is committed or after the sub-transaction is committed, so the sub-transaction service needs to Make it idempotent.

DTM provides auxiliary tables and auxiliary functions to help users quickly implement idempotency. For Mysql, he will create an auxiliary table barrier in the business database. When the user opens a transaction to adjust the balance, he will first write the gid in the barrier table. If this is a repeated request, when writing the gid, it will find that it is repeated and fails. , at this time, the balance adjustment on the user's business is skipped to ensure idempotency. The code for the helper function is as follows:

 app.POST(BusiAPI+"/SagaBTransIn", dtmutil.WrapHandler2(func(c *gin.Context) interface{} {
    return MustBarrierFromGin(c).Call(txGet(), func(tx *sql.Tx) error {
        return SagaAdjustBalance(tx, TransInUID, reqFrom(c).Amount, reqFrom(c).TransInResult)
    })
}))

The principle of Mongo's handling of idempotency is similar to that of Mysql, so I won't repeat it.

The principle of Redis dealing with idempotency is different from that of Mysql, mainly because the principle of transaction is different. Redis transactions are mainly guaranteed by lua's atomic execution. The auxiliary function of DTM will adjust the balance through the lua script. Before adjusting the balance, the gid will be queried in redis. If it exists, the balance adjustment on the business will be skipped; if it does not exist, the balance adjustment on the business will be performed. The code for the helper function is as follows:

 app.POST(BusiAPI+"/SagaRedisTransOut", dtmutil.WrapHandler2(func(c *gin.Context) interface{} {
    return MustBarrierFromGin(c).RedisCheckAdjustAmount(RedisGet(), GetRedisAccountKey(TransOutUID), -reqFrom(c).Amount, 7*86400)
}))

How to make compensation

For Saga, we also need to deal with the compensation operation, but the compensation operation is not a simple reverse adjustment, and there are many pits that need to be paid attention to, otherwise it is easy to compensate for errors.
On the one hand, compensation needs to consider idempotency, because in the compensation process, it also needs to consider the case of fault retry, just like the idempotent processing in the previous section. On the other hand, compensation also needs to consider null compensation, because the forward branch fails to return. This failure may be a failure after the forward data has been adjusted and submitted, or it may return a failure before it is submitted. For failures where data has been committed, we need to perform the reverse operation, and for failures where data is not committed, we need to skip the reverse operation, that is, handle null compensation.

In the auxiliary table and auxiliary function provided by DTM, on the one hand, it will judge whether it is empty compensation according to the gid inserted in the forward operation, and on the other hand, it will also insert gid+'compensate' to judge whether the compensation is a repeated operation. If it is a normal compensation operation, the compensation on the service will be performed, and if it is an empty compensation or repeated compensation, the compensation on the compensation service will be skipped.

Mysql's code is as follows:

 app.POST(BusiAPI+"/SagaBTransInCom", dtmutil.WrapHandler2(func(c *gin.Context) interface{} {
    return MustBarrierFromGin(c).Call(txGet(), func(tx *sql.Tx) error {
        return SagaAdjustBalance(tx, TransInUID, -reqFrom(c).Amount, "")
    })
}))

The code for Redis is as follows:

 app.POST(BusiAPI+"/SagaRedisTransOutCom", dtmutil.WrapHandler2(func(c *gin.Context) interface{} {
    return MustBarrierFromGin(c).RedisCheckAdjustAmount(RedisGet(), GetRedisAccountKey(TransOutUID), reqFrom(c).Amount, 7*86400)
}))

The compensation code is almost the same as the previous code for the forward operation, except that the amount is multiplied by -1. The auxiliary function of DTM will contain both idempotent and compensation related logic inside a function

Other exceptions

When writing sub-transactions and compensation for sub-transactions, there is actually another abnormal situation: suspension, which may occur when the global transaction rolls back over time, or rolls back after the retry arrives online. The normal situation is to first operate forward and then compensate, but in extreme cases Compensation may occur first and then forward operation. Therefore, forward operation also needs to determine whether compensation has been performed. If it has been performed, the business operation needs to be skipped.

For DTM users, these exceptions have been handled gracefully and properly. As a user, you only need to call according to the above MustBarrierFromGin(c).Call , and you don't need to care about these exceptions at all. How the DTM handles these exceptions is described in detail here: Exceptions and Subtransaction Barriers

Initiate a distributed transaction

After writing each sub-transaction service, the following part of the code initiates a Saga global transaction:

 saga := dtmcli.NewSaga(dtmutil.DefaultHTTPServer, dtmcli.MustGenGid(dtmutil.DefaultHTTPServer)).
  Add(busi.Busi+"/SagaBTransOut", busi.Busi+"/SagaBTransOutCom", &busi.TransReq{Amount: 50}).
  Add(busi.Busi+"/SagaMongoTransIn", busi.Busi+"/SagaMongoTransInCom", &busi.TransReq{Amount: 30}).
  Add(busi.Busi+"/SagaRedisTransIn", busi.Busi+"/SagaRedisTransOutIn", &busi.TransReq{Amount: 20})
err := saga.Submit()

In this part of the code, a Saga global transaction is created, which includes 3 sub-transactions:

• Transfer out 50 from Mysql
• Transfer 30 to Mongo
• Transfer 20 to Redis

During the entire transaction process, if all sub-transactions are successfully completed, the global transaction is successful; if a sub-transaction returns a business failure, then the global transaction is rolled back.

run

If you want to run the above example in full, the steps are as follows:

1. run dtm

 git clone https://github.com/dtm-labs/dtm && cd dtm
go run main.go

2. A successful example

 git clone https://github.com/dtm-labs/dtm-examples && cd dtm-examples
go run main.go http_saga_multidb

3. Example of failure to run

 git clone https://github.com/dtm-labs/dtm-examples && cd dtm-examples
go run main.go http_saga_multidb_rollback

You can modify the example to simulate various temporary failures, null compensation situations, and various other exceptions, and when the entire global transaction finally completes, the data is consistent.

summary

This article gives an example of a distributed transaction across Mysql, Redis, and Mongo, and explains in detail the problems and solutions that need to be dealt with.

The principles in this article apply to all storage engines that support ACID transactions, and you can quickly extend it for other engines, such as TiKV, etc.

Welcome to https://github.com/dtm-labs/dtm and star support us