Distributed transaction framework seata landing practice

Preface

seata is a set of open source distributed transaction framework developed by Alibaba, which provides several transaction modes of AT, TCC, SAGA and XA. This article takes the logistics background service of the boutique course project team as an example to introduce the process of landing the seata framework, the problems encountered and the solutions.

author / Deng Xinwei

edit / NetEase Youdao

Youdao Premium Class Educational Administration System is a distributed cluster service based on springcloud. In actual business, there are many distributed transaction scenarios. However, the traditional transaction framework cannot achieve global transactions. For a long time, the consistency of our distributed scenes often refers to giving up strong consistency and ensuring final consistency.

We found from the survey that the seata framework can not only meet business needs, be flexible and compatible with multiple transaction modes, but also achieve strong data consistency.

This article takes logistics business as an example, and records some problems and solutions encountered in the process of landing the seata framework in actual business, for everyone to learn and discuss~ Welcome everyone to discuss and exchange in the message area

1. Basic information

• seata version: 1.4
• Microservice framework: springcloud
• Registration Center: consul

2. Basic framework

2.1 Basic components

The seata frame is divided into 3 components:

• TC (Transaction Coordinator) -transaction coordinator (ie seata-server)

Maintain the status of global and branch transactions, and drive global transaction commit or rollback.

• TM (Transaction Manager) -Transaction Manager (on the client, the service that initiates transactions)

Define the scope of the global transaction: start the global transaction, commit or roll back the global transaction.

• RM (Resource Manager) Resource Manager (on client)

Manage the resources of branch transaction processing, talk with TC to register branch transaction and report the status of branch transaction, and drive branch transaction commit or rollback

2.2. Deploy seata-server(TC)

Download the seata server on the official website, unzip it and execute bin/seata-server.sh to start it.

seata-server has two configuration files: registry.conf and file.conf. The registry.conf file determines the registry configuration used by seata-server and the method of obtaining configuration information.

We use consul as the registry, so we need to modify the following configuration in the registry.conf file:

registry {
  #file 、nacos 、eureka、redis、zk、consul、etcd3、sofa
  type = "consul" ## 这里注册中心填consul
  loadBalance = "RandomLoadBalance"
  loadBalanceVirtualNodes = 10
   ... ...
  consul {
    cluster = "seata-server"
    serverAddr = "***注册中心地址***"
    #这里的dc指的是datacenter，若consul为多数据源配置需要在请求中加入dc参数。
    #dc与namespace并非是seata框架自带的，文章后面将会进一步解释
    dc="bj-th"
    namespace="seata-courseop"
  }
  ... ...
}

config {
  # file、nacos 、apollo、zk、consul、etcd3
  ## 如果启动时从注册中心获取基础配置信息，填consul
  ## 否则从file.conf文件中获取
  type = "consul"
  consul {
    serverAddr = "127.0.0.1:8500"
  }
... ...
}

It should be noted that if high availability deployment is required, the way seata obtains configuration information must be the registry, and file.conf is useless at this time.

(Of course, you need to migrate the configuration information in the file.conf file to consul in advance)

store {
  ## store mode: file、db、redis
  mode = "db"

... ...
  ## database store property
  ## 如果使用数据库模式，需要配置数据库连接设置
  db {
    ## the implement of javax.sql.DataSource, such as DruidDataSource(druid)/BasicDataSource(dbcp)/HikariDataSource(hikari) etc.
    datasource = "druid"
    ## mysql/oracle/postgresql/h2/oceanbase etc.
    dbType = "mysql"
    driverClassName = "com.mysql.jdbc.Driver"
    url = "jdbc:mysql://***线上数据库地址***/seata"
    user = "******"
    password = "******"
    minConn = 5
    maxConn = 100
    ## 这里的三张表需要提前在数据库建好
    globalTable = "global_table"
    branchTable = "branch_table"
    lockTable = "lock_table"
    queryLimit = 100
    maxWait = 5000
  }
... ...
}

service {
  #vgroup->rgroup
  vgroupMapping.tx-seata="seata-server"
  default.grouplist="127.0.0.1:8091"
  #degrade current not support
  enableDegrade = false
  #disable
  disable = false
  max.commit.retry.timeout = "-1"
  max.rollback.retry.timeout = "-1"
}

Among them, global_table , branch_table , lock_table three tables need to be built in the database in advance.

2.3 Configure the client side (RM and TM)

Every service that uses the seata framework needs to introduce seata components

dependencies {

    api 'com.alibaba:druid-spring-boot-starter:1.1.10'
    api 'mysql:mysql-connector-java:6.0.6'
    api('com.alibaba.cloud:spring-cloud-alibaba-seata:2.1.0.RELEASE') {
        exclude group:'io.seata', module:'seata-all'
    }
    api 'com.ecwid.consul:consul-api:1.4.5'
    api 'io.seata:seata-all:1.4.0'
}

Each service also needs to configure the file.conf and registry.conf files and place them in the resource directory. The registry.conf is consistent with the server. In the file.conf file, in addition to the db configuration, you also need to configure the client parameters:

client {
  rm {
    asyncCommitBufferLimit = 10000
    lock {
      retryInterval = 10
      retryTimes = 30
      retryPolicyBranchRollbackOnConflict = true
    }
    reportRetryCount = 5
    tableMetaCheckEnable = false
    reportSuccessEnable = false
  }
  tm {
    commitRetryCount = 5
    rollbackRetryCount = 5
  }
  undo {
    dataValidation = true
    logSerialization = "jackson"
    ## 这个undo_log也需要提前在mysql中创建
    logTable = "undo_log"
  }
  log {
    exceptionRate = 100
  }
}

Add the seata configuration in the application.yml file:

spring:
  cloud:
      seata: ## 注意tx-seata需要与服务端和客户端的配置文件保持一致
        tx-service-group: tx-seata

In addition, you need to replace the data source of the project,

@Primary
    @Bean("dataSource")
    public DataSource druidDataSource(){
        DruidDataSource druidDataSource = new DruidDataSource();
        druidDataSource.setUrl(url);
        druidDataSource.setUsername(username);
        druidDataSource.setPassword(password);
        druidDataSource.setDriverClassName(driverClassName);
        return new DataSourceProxy(druidDataSource);
    }

At this point, the client configuration has also been completed.

3. Function demonstration

A distributed global transaction is a two-phase commit model as a whole.

global transaction is composed of several branch transactions,

branch transaction must meet the two-phase commit model requirements, that is, each branch transaction needs to have its own:

• One stage prepare behavior
• Two-stage commit or rollback behavior

According to the two-phase behavior mode, we divide the branch transaction into Automatic (Branch) Transaction Mode and TCC (Branch) Transaction Mode .

3.1 AT mode

The AT mode is based on a relational database that supports local ACID transactions:

• The first stage prepare behavior: In the local transaction, the business data update and the corresponding rollback log record are submitted together.
• Behavior of the second phase commit : It ends successfully immediately, and automatically and asynchronously batch cleans up the rollback log.
• The second stage rollback behavior: through the rollback log, automatically generate compensation operations, complete data rollback

Directly add the annotation @GlobalTransactional to the method that needs to add a global transaction

  @SneakyThrows
    @GlobalTransactional
    @Transactional(rollbackFor = Exception.class)
    public void buy(int id, int itemId){
        // 先生成订单
        Order order = orderFeignDao.create(id, itemId);
        // 根据订单扣减账户余额
        accountFeignDao.draw(id, order.amount);
    }

Note: Like @Transactional, @GlobalTransactional must be satisfied in order to take effect:

• The target function must be of public type
• When the method in the same class is called, the method of calling the target function must be called in the form of springBeanName.method. You cannot use this to directly call the internal method

3.2 TCC mode

The TCC mode supports the incorporation of custom branch transactions into the management of global transactions.

• The first stage prepare behavior: call custom prepare logic.
• The second stage commit behavior: call custom commit logic.
• The second stage rollback behavior: call custom rollback logic.

First, write a TCC service interface:

@LocalTCC
public interface BusinessAction {
    @TwoPhaseBusinessAction(name = "doBusiness", commitMethod = "commit", rollbackMethod = "rollback")
    boolean doBusiness(BusinessActionContext businessActionContext,
                       @BusinessActionContextParameter(paramName = "message") String msg);

    boolean commit(BusinessActionContext businessActionContext);

    boolean rollback(BusinessActionContext businessActionContext);
}

which , BusinessActionContext is the global transaction context, you can get the global transaction related information from this object (if it is the initiator of the global transaction, it will be automatically generated after passing in null), and then implement this interface:

@Slf4j
@Service
public class BusinessActionImpl implements BusinessAction {

    @Transactional(rollbackFor = Exception.class)
    @Override
    public boolean doBusiness(BusinessActionContext businessActionContext, String msg) {
        log.info("准备do business:{}",msg);
        return true;
    }

    @Transactional(rollbackFor = Exception.class)
    @Override
    public boolean commit(BusinessActionContext businessActionContext) {
        log.info("business已经commit");
        return true;
    }

    @Transactional(rollbackFor = Exception.class)
    @Override
    public boolean rollback(BusinessActionContext businessActionContext) {
        log.info("business已经rollback");
        return true;
    }
}

finally , the method of opening the global transaction is the same as AT mode.

@SneakyThrows
    @GlobalTransactional
    public void doBusiness(BusinessActionContext context, String msg){
        accountFeignDao.draw(3, new BigDecimal(100));
        businessAction.doBusiness(context, msg);
    }

4. Problems encountered

4.1 Client TM/RM cannot register to TC

When deploying the seata project, we often encounter such a problem: everything is normal when debugging locally, but when trying to deploy to the line, the clinet side always prompts the registration of the TC side to fail.

• This is because the client needs to discover through the service first, find the service information of the seata-server in the registry, and then establish a connection with the seata-server. However, the online consul adopts a multi-data center model. When calling the consul api, the dc parameter item must be added, otherwise the correct service information will not be returned; however, the consul service discovery component provided by seata does not seem to support the dc parameter. Configuration.
• There is another reason why the client cannot connect to the TC: seata's consul client uses the wait and index parameters when calling the service status monitoring api, so that the consul query enters the blocking query mode. At this time, the client monitors the key to be queried in consul, and returns the result only when the key changes or the maximum request time is reached. It seems that due to the problem of the consul version, this blocking query does not monitor the change of the key, but instead causes the thread discovered by the service to fall into infinite waiting, and naturally it is impossible for the client to obtain the registration information of the server.

4.2 High-availability deployment

The highly available deployment of seata service only supports the registry mode . Therefore, we need to find a way to save the file.conf file in consul in the form of key-value pairs.

Unfortunately, consul does not explicitly support namespace. We can only use "/" as the separator in the put request to achieve a similar effect. Of course, the seata framework does not take this into account. So we need to modify the Consul implementation class of the Configuration interface and the RegistryProvider interface in the source code to add the namespace attribute

4.3 global_log and branch_log

When TC wants to insert log data into mysql, it occasionally reports:

Caused by: java.sql.SQLException: Incorrect string value:

The application_data field is actually a record of business data. The official statement to build a table is as follows:

CREATE TABLE IF NOT EXISTS `global_table`
(
    `xid`                       VARCHAR(128) NOT NULL,
    `transaction_id`            BIGINT,
    `status`                    TINYINT      NOT NULL,
    `application_id`            VARCHAR(32),
    `transaction_service_group` VARCHAR(32),
    `transaction_name`          VARCHAR(128),
    `timeout`                   INT,
    `begin_time`                BIGINT,
    `application_data`          VARCHAR(2000),
    `gmt_create`                DATETIME,
    `gmt_modified`              DATETIME,
    PRIMARY KEY (`xid`),
    KEY `idx_gmt_modified_status` (`gmt_modified`, `status`),
    KEY `idx_transaction_id` (`transaction_id`)
) ENGINE = InnoDB
  DEFAULT CHARSET = utf8;

Obviously, the size of VARCHAR(2000) is inappropriate, and the format of utf8 is also inappropriate. So we need to modify part of the code of data source connection

// connectionInitSql设置
    protected Set<String> getConnectionInitSqls(){
        Set<String> set = new HashSet<>();
        String connectionInitSqls = CONFIG.getConfig(ConfigurationKeys.STORE_DB_CONNECTION_INIT_SQLS);
        if(StringUtils.isNotEmpty(connectionInitSqls)) {
            String[] strs = connectionInitSqls.split(",");
            for(String s:strs){
                set.add(s);
            }
        }
        // 默认支持utf8mb4
        set.add("set names utf8mb4");
        return set;
    }

5. Custom development

5.1 Use SPI mechanism to write custom components

Seata provides the function of customizing the implementation interface based on the spi mechanism of java. We only need to write our own implementation class according to the interface of seata in our own service.

SPI (Service Provider Interface) is a built-in service discovery mechanism of JDK. It is used to call services through interfaces between different modules to avoid coupling of specific implementation classes of specific service interfaces. For example, in the JDBC database driver module, different database connection drivers have the same interface but different implementation classes. Before using the SPI mechanism, the driver code needs to be called directly in the class in the form of Class.forName (full name of the specific implementation class), so that the caller depends on In order to realize the specific driver, the code should be modified when the driver is replaced.

Take ConsulRegistryProvider as an example:

• ConsulRegistryServiceImpl

  
  // 增加DC和namespace
    private static String NAMESPACE;
    private static String DC;
  
    private ConsulConfiguration() {
        Config registryCongig = ConfigFactory.parseResources("registry.conf");
        NAMESPACE = registryCongig.getString("config.consul.namespace");
        DC = CommonSeataConfiguration.getDatacenter();
        consulNotifierExecutor = new ThreadPoolExecutor(THREAD_POOL_NUM, THREAD_POOL_NUM, Integer.MAX_VALUE,
                TimeUnit.MILLISECONDS, new LinkedBlockingQueue<>(),
                new NamedThreadFactory("consul-config-executor", THREAD_POOL_NUM));
    }
    ... ...
  // 同时在getHealthyServices中，删除请求参数wait&index    
    /**
     * get healthy services
     *
     * @param service
     * @return
     */
    private Response<List<HealthService>> getHealthyServices(String service, long index, long watchTimeout) {
        return getConsulClient().getHealthServices(service, HealthServicesRequest.newBuilder()
                .setTag(SERVICE_TAG)
                .setDatacenter(DC)
                .setPassing(true)
                .build());
    }

• ConsulRegistryProvider Note that the order must be greater than the default value 1 in the seata package, and the seata class loader will first load implementation classes with a larger order

  @LoadLevel(name = "Consul" ,order = 2)
  public class ConsulRegistryProvider implements RegistryProvider {
   @Override
   public RegistryService provide() {
       return ConsulRegistryServiceImpl.getInstance();
   }
  }

• Then add in the services directory of META-INF: io.seata.discovery.registry.RegistryProvider

  
  com.youdao.ke.courseop.common.seata.ConsulRegistryProvider
  

  This can replace the implementation in the seata package.

5.2 common-seata toolkit

For these custom implementation classes, as well as some public client configurations, we can package them into a toolkit:

In this way, other projects only need to import this toolkit, and can be used directly without tedious configuration.

Gradle introduces the common package:

api 'com.youdao.ke.courseop.common:common-seata:0.0.+'

6. Landing examples

Take a logistics scenario as an example:
Business Architecture :

• logistics-server (logistics service)
• logistics-k3c-server (Logistics-Kingdee client, encapsulating API for calling Kingdee service
• elasticsearch

Business background : logistics execution requisition is added, data is updated in elasticsearch, and Kingdee outbound method of logistics-k3c is called through rpc to generate Kingdee receipts, as shown in Figure 2

problem : If the elasticsearch receipt update is abnormal, the Kingdee receipt cannot be rolled back, causing data inconsistency.

After deploying the seata online service, you only need to introduce the common-seata toolkit in logistics and logistics-k3c respectively

logistics service :

 // 使用全局事务注解开启全局事务
    @GlobalTransactional
    @Transactional(rollbackFor = Exception.class)
    public void Scm通过(StaffOutStockDoc staffOutStock, String body) throws Exception {
        ... 一些业务处理...
         // 构建金蝶单据请求
        K3cApi.StaffoutstockReq req = new K3cApi.StaffoutstockReq();
        req.materialNums = materialNums;
        req.staffOutStockId = staffOutStock.id;
        ... 一些业务处理 ...
       // 调用logistics-k3c-api金蝶出库
        k3cApi.staffoutstockAuditPass(req);

        staffOutStock.status = 待发货;
        staffOutStock.scmAuditTime = new Date();
        staffOutStock.updateTime = new Date();
        staffOutStock.historyPush("scm通过");
        // 更新对象后存入elasticsearch
        es.set(staffOutStock);
    }

logistics-k3c：

Since our newly-added document interface is to call Kingdee’s services, we use the TCC mode to construct a transaction interface here.

• First create the StaffoutstockCreateAction interface

  @LocalTCC
  public interface StaffoutstockCreateAction {
    @TwoPhaseBusinessAction(name = "staffoutstockCreate")
    boolean create(BusinessActionContext businessActionContext,
                       @BusinessActionContextParameter(paramName = "staffOutStock") StaffOutStock staffOutStock,
                       @BusinessActionContextParameter(paramName = "materialNum") List<Triple<Integer, Integer, Integer>> materialNum);
  
    boolean commit(BusinessActionContext businessActionContext);
  
    boolean rollback(BusinessActionContext businessActionContext);
  
  }

• Interface implements StaffoutstockCreateActionImpl

  @Slf4j
  @Service
  public class StaffoutstockCreateActionImpl implements StaffoutstockCreateAction {
  
    @Autowired
    private K3cAction4Staffoutstock k3cAction4Staffoutstock;
  
    @SneakyThrows
    @Transactional(rollbackFor = Exception.class)
    @Override
    public boolean create(BusinessActionContext businessActionContext, StaffOutStock staffOutStock, List<Triple<Integer, Integer, Integer>> materialNum) {
        //金蝶单据新增
        k3cAction4Staffoutstock.staffoutstockAuditPass(staffOutStock, materialNum);
        return true;
    }
  
    @SneakyThrows
    @Transactional(rollbackFor = Exception.class)
    @Override
    public boolean commit(BusinessActionContext businessActionContext) {
        Map<String, Object> context = businessActionContext.getActionContext();
        JSONObject staffOutStockJson = (JSONObject) context.get("staffOutStock");
        // 如果尝试新增成功，commit不做任何事
        StaffOutStock staffOutStock = staffOutStockJson.toJavaObject(StaffOutStock.class);
        log.info("staffoutstock {} commit successfully!", staffOutStock.id);
        return true;
    }
  
    @SneakyThrows
    @Transactional(rollbackFor = Exception.class)
    @Override
    public boolean rollback(BusinessActionContext businessActionContext) {
        Map<String, Object> context = businessActionContext.getActionContext();
        JSONObject staffOutStockJson = (JSONObject) context.get("staffOutStock");
        StaffOutStock staffOutStock = staffOutStockJson.toJavaObject(StaffOutStock.class);
        // 这里调用金蝶单据删除接口进行回滚
        k3cAction4Staffoutstock.staffoutstockRollback(staffOutStock);
        log.info("staffoutstock {} rollback successfully!", staffOutStock.id);
        return true;
    }
  }
  

• Encapsulation as a business method

  /**
     * 项目组领用&报废的审核通过：新增其他出库单
     * 该方法使用seata-TCC方案实现全局事务
     * @param staffOutStock
     * @param materialNum
     */
    
    @Transactional
    public void staffoutstockAuditPassWithTranscation(StaffOutStock staffOutStock,
                                                      List<Triple<Integer, Integer, Integer>> materialNum){
        staffoutstockCreateAction.create(null, staffOutStock, materialNum);
    }

• k3c API implementation class

  
   @SneakyThrows
    @Override
    public void staffoutstockAuditPass(StaffoutstockReq req) {
        ... 一些业务处理方法 ...
        //这里调用了封装好的事务方法
        k3cAction4Staffoutstock.staffoutstockAuditPassWithTranscation(staffOutStock, triples);
    }

In this way, a TCC-based global transaction link is established.

When the global transaction successfully executed and , we can see the printed log in the server (Figure 3):

If the global transaction fails to execute , it will be rolled back. At this time, the rollback in the interface will be executed, and the Kingdee interface will be called to delete the generated documents, as shown in Figure 4.

7. Summary

In this paper, deployment and use seata framework for main , recorded seata framework the use of some of key steps and technical details , and provides solutions to some of the technical problems encountered during the project landing.

In subsequent tweets, we will continue to use the source code analysis of the seata framework as the main line to introduce to you the core principles and technical details of seata to implement distributed transactions.
-END-