What does online distributed transaction look like? Take python's saga as an example

Bank inter-bank transfer business is a typical distributed transaction scenario. Assuming that A needs to transfer funds across banks to B, then the data of two banks is involved. The ACID of the transfer cannot be guaranteed by the local transaction of a database, and it can only be solved by distributed transactions.

Distributed transaction

Distributed transactions in a distributed environment, in order to meet the needs of availability, performance and degraded services, and reduce the requirements of consistency and isolation, on the one hand, follow the BASE theory:

• Basic Availability (Basic Availability)
• Soft state
• Eventual consistency (Eventual consistency)

On the other hand, distributed transactions also partially follow the ACID specification:

• Atomicity: strictly follow
• Consistency: The consistency after the transaction is completed is strictly followed; the consistency in the transaction can be appropriately relaxed
• Isolation: Parallel transactions cannot be affected; the visibility of the intermediate results of the transaction allows security relaxation
• Persistence: Strictly follow

SAGA

Saga is a distributed transaction scheme mentioned in this database paper SAGAS The core idea is to split the long transaction into multiple local short transactions, which are coordinated by the Saga transaction coordinator. If each local transaction is successfully completed, it will be completed normally. If a step fails, the compensation operation will be called once in the reverse order.

The open source framework currently available for SAGA is mainly the Java language, with seata being the representative. Our example uses the go language, and the distributed transaction framework used is https://github.com/yedf/dtm , which supports distributed transactions very elegantly. Let's explain in detail the composition of SAGA:

In the DTM transaction framework, there are 3 roles, the same as the classic XA distributed transaction:

• AP/application, initiate a global transaction, define which transaction branches the global transaction contains
• RM/resource manager, responsible for the management of various resources of branch affairs
• TM/transaction manager, responsible for coordinating the correct execution of global transactions, including the execution of SAGA forward/reverse operations

Looking at a successfully completed SAGA sequence diagram below, it is easy to understand SAGA distributed transactions:

SAGA practice

For the example of the bank transfer we want to perform, we will perform the transfer in and out in the forward operation, and make the opposite adjustment in the compensation operation.

First we create the account balance table:

CREATE TABLE dtm_busi.`user_account` (
  `id` int(11) AUTO_INCREMENT PRIMARY KEY,
  `user_id` int(11) not NULL UNIQUE ,
  `balance` decimal(10,2) NOT NULL DEFAULT '0.00',
  `create_time` datetime DEFAULT now(),
  `update_time` datetime DEFAULT now()
);

We first write the core business code to adjust the user’s account balance

def saga_adjust_balance(cursor, uid, amount):
  affected = utils.sqlexec(cursor, "update dtm_busi.user_account set balance=balance+%d where user_id=%d and balance >= -%d" %(amount, uid, amount))
  if affected == 0:
    raise Exception("update error, balance not enough")

Let's write the processing function of the specific forward operation/compensation operation

@app.post("/api/TransOutSaga")
def trans_out_saga():
  saga_adjust_balance(c, out_uid, -30)
  return {"dtm_result": "SUCCESS"}

@app.post("/api/TransOutCompensate")
def trans_out_compensate():
  saga_adjust_balance(c, out_uid, 30)
  return {"dtm_result": "SUCCESS"}

@app.post("/api/TransInSaga")
def trans_in_saga():
  saga_adjust_balance(c, in_uid, 30)
  return {"dtm_result": "SUCCESS"}

@app.post("/api/TransInCompensate")
def trans_in_compensate():
  saga_adjust_balance(c, in_uid, -30)
  return {"dtm_result": "SUCCESS"}

At this point, the processing function of each sub-transaction has been OK, and then the SAGA transaction is opened, and the branch call is made

# 这是dtm服务地址
dtm = "http://localhost:8080/api/dtmsvr"
# 这是业务微服务地址
svc = "http://localhost:5000/api"

    req = {"amount": 30}
    s = saga.Saga(dtm, utils.gen_gid(dtm))
    s.add(req, svc + "/TransOutSaga", svc + "/TransOutCompensate")
    s.add(req, svc + "/TransInSaga", svc + "/TransInCompensate")
    s.submit()

At this point, a complete SAGA distributed transaction has been written.

If you want to run a successful example completely, then refer to this example yedf/dtmcli-py-sample , it is very simple to run it

# 部署启动dtm
# 需要docker版本18以上
git clone https://github.com/yedf/dtm
cd dtm
docker-compose up

# 另起一个命令行
git clone https://github.com/yedf/dtmcli-py-sample
cd dtmcli-py-sample
pip3 install flask dtmcli requests
flask run

# 另起一个命令行
curl localhost:5000/api/fireSaga

Handling network exceptions

Suppose that in the transaction submitted to dtm, when the transfer operation is called, there is a short-term failure, what should I do? According to the SAGA transaction agreement, dtm will retry unfinished operations. What should we do at this time? The fault may be a network failure after the transfer operation is completed, or a machine downtime during the transfer operation. How to deal with it to ensure that the account balance adjustment is correct and problem-free?

Proper handling of such network exceptions is a big problem in distributed transactions. There are three types of exceptions: repeated requests, empty compensation, and suspension, all of which need to be handled correctly

DTM provides a sub-transaction barrier function to ensure that the business logic under the above abnormal conditions will only be successfully submitted once in the correct order. (Sub-barrier transaction details Reference Distributed Transaction most classic seven solutions child affairs barrier links)

We adjust the processing function to:

@app.post("/api/TransOutSaga")
def trans_out_saga():
  with barrier.AutoCursor(conn_new()) as cursor:
    def busi_callback(c):
      saga_adjust_balance(c, out_uid, -30)
    barrier_from_req(request).call(cursor, busi_callback)
  return {"dtm_result": "SUCCESS"}

The barrier_from_req(request).call(cursor, busi_callback) call here uses sub-transaction barrier technology to ensure that the busi_callback callback function is only submitted once​

You can try to call this TransIn service multiple times, with only one balance adjustment.

Handle rollback

What happens if the bank finds that the account of user 2 is abnormal when the bank is about to transfer the amount to user 2 and the return fails? We adjust the processing function so that the transfer operation returns to fail

@app.post("/api/TransInSaga")
def trans_in_saga():
  return {"dtm_result": "FAILURE"}

We give a sequence diagram of transaction failure interaction

There is one point here. The forward operation of TransIn did not do anything, and it returned a failure. At this time, calling the compensation operation of TransIn would cause the reverse adjustment to go wrong?

Don't worry, the previous sub-transaction barrier technology can ensure that if a TransIn error occurs before the submission, the compensation is a no-op; if the TransIn error occurs after the submission, the compensation operation will submit the data once.

You can change the TransIn that returns an error to:

@app.post("/api/TransInSaga")
def trans_in_saga():
  with barrier.AutoCursor(conn_new()) as cursor:
    def busi_callback(c):
      saga_adjust_balance(c, in_uid, 30)
    barrier_from_req(request).call(cursor, busi_callback)
  return {"dtm_result": "FAILURE"}

The final result balance will still be correct, the principle can be referred to: most classic seven solutions for distributed transactions Sub-transaction barrier link

summary

In this article, we introduced the theoretical knowledge of SAGA, and through an example, gave a complete process of writing a SAGA transaction, covering normal successful completion, abnormal situations, and successful rollback situations. I believe that readers have a deep understanding of SAGA through this article.

The dtm used in this article is a new open source Golang distributed transaction management framework with powerful functions. It supports transaction modes such as TCC, SAGA, XA, transaction messages, and supports languages ​​such as Go, python, PHP, node, and csharp. It also provides a very simple and easy-to-use interface.

After reading this dry goods, welcome everyone to visit project https://github.com/yedf/dtm , give stars to support!