How Spring supports multiple data sources

Hello everyone, I'm the class representative.
Welcome to my public account: Java class representative .

The first part introduced the basics of data sources, and implemented multiple data sources based on two sets of DataSource and two sets of mybatis configurations, and explained the realization idea of multiple data sources from the basic knowledge level. For those who don’t know, please poke→ Student, your multi-data source transaction is invalid!

As mentioned in the review at the end of the article, this method of multiple data sources is very intrusive to the code, and each component needs to be written in two sets, which is not suitable for large-scale online practice.

For the requirement of multiple data sources, Spring noticed and gave a solution as early as 2007, see the original text: dynamic-datasource-routing

Spring provides a AbstractRoutingDataSource class, which is used to implement on-demand routing of multiple DataSource . This article introduces the practice of multiple data sources based on this method. .

1. What is AbstractRoutingDataSource

First look at the annotations on the class:

Abstract {@link javax.sql.DataSource} implementation that routes {@link #getConnection()}
The latter is usually calls to one of various target DataSources based on a lookup key.
(but not necessarily) determined through some thread-bound transaction context.

Class represents translation: This is an abstract class that can route calls to the getConnection() method to the target DataSource lookup key The latter (referring to lookup key ) is usually determined by the context bound to the thread.

This comment can be described as every word, not a single nonsense. The meaning is explained below in conjunction with the main code.

 public abstract class AbstractRoutingDataSource extends AbstractDataSource implements InitializingBean {

    //目标 DataSource Map，可以装很多个 DataSource
    @Nullable
    private Map<Object, Object> targetDataSources;
    
    @Nullable
    private Map<Object, DataSource> resolvedDataSources;

    //Bean初始化时，将 targetDataSources 遍历并解析后放入 resolvedDataSources
    @Override
    public void afterPropertiesSet() {
        if (this.targetDataSources == null) {
            throw new IllegalArgumentException("Property 'targetDataSources' is required");
        }
        this.resolvedDataSources = CollectionUtils.newHashMap(this.targetDataSources.size());
        this.targetDataSources.forEach((key, value) -> {
            Object lookupKey = resolveSpecifiedLookupKey(key);
            DataSource dataSource = resolveSpecifiedDataSource(value);
            this.resolvedDataSources.put(lookupKey, dataSource);
        });
        if (this.defaultTargetDataSource != null) {
            this.resolvedDefaultDataSource = resolveSpecifiedDataSource(this.defaultTargetDataSource);
        }
    }
    
    @Override
    public Connection getConnection() throws SQLException {
        return determineTargetDataSource().getConnection();
    }

    /**
     * Retrieve the current target DataSource. Determines the
     * {@link #determineCurrentLookupKey() current lookup key}, performs
     * a lookup in the {@link #setTargetDataSources targetDataSources} map,
     * falls back to the specified
     * {@link #setDefaultTargetDataSource default target DataSource} if necessary.
     * @see #determineCurrentLookupKey()
     */
     //根据 #determineCurrentLookupKey()返回的lookup key 去解析好的数据源 Map 里取相应的数据源
    protected DataSource determineTargetDataSource() {
        Assert.notNull(this.resolvedDataSources, "DataSource router not initialized");
        // 当前 lookupKey 的值由用户自己实现↓
        Object lookupKey = determineCurrentLookupKey();
        DataSource dataSource = this.resolvedDataSources.get(lookupKey);
        if (dataSource == null && (this.lenientFallback || lookupKey == null)) {
            dataSource = this.resolvedDefaultDataSource;
        }
        if (dataSource == null) {
            throw new IllegalStateException("Cannot determine target DataSource for lookup key [" + lookupKey + "]");
        }
        return dataSource;
    }
    
    /**
     * Determine the current lookup key. This will typically be
     * implemented to check a thread-bound transaction context.
     * <p>Allows for arbitrary keys. The returned key needs
     * to match the stored lookup key type, as resolved by the
     * {@link #resolveSpecifiedLookupKey} method.
     */
    // 该方法用来决定lookup key，通常用线程绑定的上下文来实现
    @Nullable
    protected abstract Object determineCurrentLookupKey();
    
    // 省略其余代码...

}

First look at the class diagram

It is DataSource and implements InitializingBean , indicating that there is an initialization operation of Bean .

Second look at instance variables

private Map<Object, Object> targetDataSources; and private Map<Object, DataSource> resolvedDataSources; are actually the same thing, the latter is obtained by parsing the former, and is essentially used to store multiple instances of DataSource Map .

Finally, look at the core method

Using DataSource , the essence is to call its getConnection() method to obtain a connection to perform database operations.

AbstractRoutingDataSource#getConnection() method first calls determineTargetDataSource() , decides which target data source to use, and uses the getConnection() of the data source to connect to the database:

 @Override
public Connection getConnection() throws SQLException {
   return determineTargetDataSource().getConnection();
}

 protected DataSource determineTargetDataSource() {
   Assert.notNull(this.resolvedDataSources, "DataSource router not initialized");
   // 这里使用的 lookupKey 就能决定返回的数据源是哪个
   Object lookupKey = determineCurrentLookupKey();
   DataSource dataSource = this.resolvedDataSources.get(lookupKey);
   if (dataSource == null && (this.lenientFallback || lookupKey == null)) {
      dataSource = this.resolvedDefaultDataSource;
   }
   if (dataSource == null) {
      throw new IllegalStateException("Cannot determine target DataSource for lookup key [" + lookupKey + "]");
   }
   return dataSource;
}

So the key point is the determineCurrentLookupKey() method, which is an abstract method, implemented by the user, by changing its return value to control the return of different data sources. Expressed in a table as follows:

lookupKey	DataSource
first	firstDataSource
second	secondDataSource

How to implement this method? Combined with Spring the tips given in the comments:

The latter (referring to lookup key ) is usually determined by the context bound to the thread.

Should be able to think of ThreadLocal now! ThreadLocal can maintain a variable bound to the current thread, acting as the context of this thread.

2. Realization

Design yaml File externalization to configure multiple data sources

 spring:
  datasource:
    first:
      driver-class-name: org.h2.Driver
      jdbc-url: jdbc:h2:mem:db1
      username: sa
      password:
    second:
      driver-class-name: org.h2.Driver
      jdbc-url: jdbc:h2:mem:db2
      username: sa
      password:

Create a context holding class for lookupKey :

 /**
 * 数据源 key 上下文
 * 通过控制 ThreadLocal变量 LOOKUP_KEY_HOLDER 的值用于控制数据源切换
 * @see RoutingDataSource
 * @author :Java课代表
 */
public class RoutingDataSourceContext {

    private static final ThreadLocal<String> LOOKUP_KEY_HOLDER = new ThreadLocal<>();

    public static void setRoutingKey(String routingKey) {
        LOOKUP_KEY_HOLDER.set(routingKey);
    }

    public static String getRoutingKey() {
        String key = LOOKUP_KEY_HOLDER.get();
        // 默认返回 key 为 first 的数据源
        return key == null ? "first" : key;
    }

    public static void reset() {
        LOOKUP_KEY_HOLDER.remove();
    }
}

Implementation AbstractRoutingDataSource :

 /**
 * 支持动态切换的数据源
 * 通过重写 determineCurrentLookupKey 实现数据源切换
 * @author :Java课代表
 */
public class RoutingDataSource extends AbstractRoutingDataSource {

    @Override
    protected Object determineCurrentLookupKey() {
        return RoutingDataSourceContext.getRoutingKey();
    }

}

For us RoutingDataSource initialize multiple data sources:

 /**
 * 数据源配置
 * 把多个数据源，装配到一个 RoutingDataSource 里
 * @author :Java课代表
 */
@Configuration
public class RoutingDataSourcesConfig {

    @Bean
    @ConfigurationProperties(prefix = "spring.datasource.first")
    public DataSource firstDataSource() {
        return DataSourceBuilder.create().build();
    }

    @Bean
    @ConfigurationProperties(prefix = "spring.datasource.second")
    public DataSource secondDataSource() {
        return DataSourceBuilder.create().build();
    }

    @Primary
    @Bean
    public RoutingDataSource routingDataSource() {
        RoutingDataSource routingDataSource = new RoutingDataSource();
        routingDataSource.setDefaultTargetDataSource(firstDataSource());
        Map<Object, Object> dataSourceMap = new HashMap<>();
        dataSourceMap.put("first", firstDataSource());
        dataSourceMap.put("second", secondDataSource());
        routingDataSource.setTargetDataSources(dataSourceMap);
        return routingDataSource;
    }

}

Demonstrate the code for manual switching:

 public void init() {
    // 手工切换为数据源 first，初始化表
    RoutingDataSourceContext.setRoutingKey("first");
    createTableUser();
    RoutingDataSourceContext.reset();

    // 手工切换为数据源 second，初始化表
    RoutingDataSourceContext.setRoutingKey("second");
    createTableUser();
    RoutingDataSourceContext.reset();

}

In this way, the most basic multi-data source switching is realized.

It is not difficult to find that the switching work can obviously be divided into a section. We can optimize it and use annotations to indicate the cut points and where to cut.

3. Introduction AOP

custom annotation

 /**
 * @author :Java课代表
 */
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface WithDataSource {
    String value() default "";
}

Create a slice

 @Aspect
@Component
// 指定优先级高于@Transactional的默认优先级
// 从而保证先切换数据源再进行事务操作
@Order(Ordered.LOWEST_PRECEDENCE - 1)
public class DataSourceAspect {

    @Around("@annotation(withDataSource)")
    public Object switchDataSource(ProceedingJoinPoint pjp, WithDataSource withDataSource) throws Throwable {

        // 1.获取 @WithDataSource 注解中指定的数据源
        String routingKey = withDataSource.value();
        // 2.设置数据源上下文
        RoutingDataSourceContext.setRoutingKey(routingKey);
        // 3.使用设定好的数据源处理业务
        try {
            return pjp.proceed();
        } finally {
            // 4.清空数据源上下文
            RoutingDataSourceContext.reset();
        }
    }
}

With annotations and facets, it's much easier to use:

 // 注解标明使用"second"数据源
@WithDataSource("second")
public List<User> getAllUsersFromSecond() {
    List<User> users = userService.selectAll();
    return users;
}

There are two details to note about facets:

1. Need to specify priority over declarative transactions

   Reason: The essence of declarative transaction transaction is also AOP, which only takes effect on the data source used when it is opened, so it must be opened after switching to the specified data source. The default priority of declarative transaction is the lowest level, here only need to set The custom data source aspect can be given a higher priority than it.

2. After the business is executed, the context must be cleared

   Reason: Suppose method A uses @WithDataSource("second") to specify the "second" data source, followed by method B without writing annotations, expecting to use the default first data source. However, because the method A put into the context lookupKey is still "second" and has not been deleted, the data source executed by method B does not match the expectation.

4. Review

So far, the multi-data source based on AbstractRoutingDataSource + AOP has been realized.

DataSource 02d203fd95e72d6e33795eb33d95a84d---这个Bean的时候，用的是自定义的RoutingDataSource ， @Primary 。 mybatis-spring-boot-starter RoutingDataSource好mybatis ，比搞DataSource +两套Mybatis The configuration scheme is much simpler.

The relevant code in the article has been uploaded to the github of the class representative

Special Note:

In the example, in order to reduce the code level and make the display more intuitive, transaction annotations are written in the controller layer. In actual development, don't do this. The task of the controller layer is to bind and verify parameters, and encapsulate the returned results. Try not to write in it. business!

5. Optimization

For general multi-data source usage scenarios, the solution in this paper has enough coverage to achieve flexible switching.

But there are still the following shortcomings:

• Relevant classes must be added when each application is used, and a lot of repetitive code
• When modifying or adding features, all related applications must be modified
• The functions are not powerful enough, and there are no advanced functions, such as load balancing of reading multiple slave libraries in the scenario of read-write separation

In fact, these codes are encapsulated into one starter , and the advanced functions can be extended slowly.

Fortunately, there are already ready-made tools in the open source world. The "baomidou" team that developed mybatis-plus has open sourced a multi-data source framework Dynamic-Datasource in its ecosystem. The underlying principle is AbstractRoutingDataSource , adding more powerful extension functions, the next part will introduce its use.