Talk about how to use the pipeline mode for business orchestration (Part 1)

foreword

1. What is the pipeline mode

The pipeline pattern is not one of the 23 design patterns we often say, it can be regarded as a variant of the chain of responsibility pattern. In technical terms, the so-called pipeline mode is to transfer data to a task queue, and the task queue processes the data in sequence.

2. What kind of scenarios are suitable for using pipeline mode

When the business process is complex, it needs to be split into multiple sub-steps, and each sub-step can be freely combined, replaced, added, or deleted.

General routines for implementing pipelines

1. Encapsulated pipeline data transparent transmission context

 public class ChannelHandlerContext extends ConcurrentHashMap<String,Object> {

    protected static Class<? extends ChannelHandlerContext> contextClass = ChannelHandlerContext.class;

    protected static final TransmittableThreadLocal<? extends ChannelHandlerContext> CHAIN_CONTEXT = new TransmittableThreadLocal<ChannelHandlerContext>() {
        @Override
        protected ChannelHandlerContext initialValue() {
            try {
                return contextClass.getDeclaredConstructor().newInstance();
            } catch (Throwable e) {
                throw new RuntimeException(e);
            }
        }
    };

    /**
     * 覆盖默认的管道上下文
     *
     * @param clazz
     */
    public static void setContextClass(Class<? extends ChannelHandlerContext> clazz) {
        contextClass = clazz;
    }

    /**
     * 获取当前管道上下文
     *
     *
     */
    public static final ChannelHandlerContext getCurrentContext() {
        return CHAIN_CONTEXT.get();
    }

    /**
     * 释放上下文资源
     *
     * @return
     */
    public void release() {
        this.clear();
        CHAIN_CONTEXT.remove();
    }

    /**
     *
     * 获取上下文默认值
     * @param key
     * @param defaultValue
     * @return
     */
    public Object getDefault(String key, Object defaultValue) {
        return Optional.ofNullable(get(key)).orElse(defaultValue);
    }

    public static final String CHANNEL_HANDLER_REQUEST_KEY = "channelHandlerRequest";

    public ChannelHandlerRequest getChannelHandlerRequest() {
        return (ChannelHandlerRequest) this.getDefault(CHANNEL_HANDLER_REQUEST_KEY,ChannelHandlerRequest.builder().build());
    }


}

2. Define the pipeline abstract executor

 public abstract class AbstactChannelHandler {

    private String channelHandlerName;

    public String getChannelHandlerName() {
        return channelHandlerName;
    }

    public void setChannelHandlerName(String channelHandlerName) {
        this.channelHandlerName = channelHandlerName;
    }

    public abstract boolean handler(ChannelHandlerContext chx);


}

3. Define the pipeline

 @Slf4j
public class ChannelPipeline {

    private LinkedBlockingDeque<AbstactChannelHandler> channelHandlers = new LinkedBlockingDeque();

    private ChannelHandlerContext handlerContext;


    public ChannelPipeline addFirst(AbstactChannelHandler channelHandler){
       return addFirst(null,channelHandler);
    }

    public ChannelPipeline addLast(AbstactChannelHandler channelHandler){
      return addLast(null,channelHandler);
    }

    public ChannelPipeline addFirst(String channelHandlerName,AbstactChannelHandler channelHandler){
        if(StringUtils.isNotBlank(channelHandlerName)){
            channelHandler.setChannelHandlerName(channelHandlerName);
        }
        channelHandlers.addFirst(channelHandler);
        return this;
    }

    public ChannelPipeline addLast(String channelHandlerName,AbstactChannelHandler channelHandler){
        if(org.apache.commons.lang3.StringUtils.isNotBlank(channelHandlerName)){
            channelHandler.setChannelHandlerName(channelHandlerName);
        }
        channelHandlers.addLast(channelHandler);
        return this;
    }


    public void setChannelHandlers(LinkedBlockingDeque<AbstactChannelHandler> channelHandlers) {
        this.channelHandlers = channelHandlers;
    }

    public ChannelHandlerContext getHandlerContext() {
        return handlerContext;
    }

    public void setHandlerContext(ChannelHandlerContext handlerContext) {
        this.handlerContext = handlerContext;
    }

    public boolean start(ChannelHandlerRequest channelHandlerRequest){
         if(channelHandlers.isEmpty()){
             log.warn("channelHandlers is empty");
             return false;
         }

        return handler(channelHandlerRequest);
    }

    private boolean handler(ChannelHandlerRequest channelHandlerRequest) {
        if(StringUtils.isBlank(channelHandlerRequest.getRequestId())){
            channelHandlerRequest.setRequestId(String.valueOf(SnowflakeUtils.getNextId()));
        }
        handlerContext.put(ChannelHandlerContext.CHANNEL_HANDLER_REQUEST_KEY,channelHandlerRequest);
        boolean isSuccess = true;
        try {
            for (AbstactChannelHandler channelHandler : channelHandlers) {
                  isSuccess = channelHandler.handler(handlerContext);
                if(!isSuccess){
                    break;
                }
            }

            if(!isSuccess){
                channelHandlers.clear();
            }
        } catch (Exception e) {
            log.error("{}",e.getMessage());
            isSuccess = false;
        } finally {
            handlerContext.release();
        }
        return isSuccess;
    }

}

4. Split different subtask pipeline executors according to the complexity of the business

 @Slf4j
public class UserCheckChannelHandler extends AbstactChannelHandler {

    
    @Override
    public boolean handler(ChannelHandlerContext chx) {
        ChannelHandlerRequest channelHandlerRequest = chx.getChannelHandlerRequest();
        System.out.println("------------------------------------步骤一：用户数据校验【"+channelHandlerRequest.getRequestId()+"】");
        Object params = channelHandlerRequest.getParams();
        if(params instanceof User){
            User user = (User)params;
            if(StringUtils.isBlank(user.getFullname())){
                log.error("用户名不能为空");
                return false;
            }
            return true;
        }


        return false;
    }
}

 @Slf4j
public class UserFillUsernameAndEmailChannelHandler extends AbstactChannelHandler {
    @SneakyThrows
    @Override
    public boolean handler(ChannelHandlerContext chx) {
        ChannelHandlerRequest channelHandlerRequest = chx.getChannelHandlerRequest();
        System.out.println("------------------------------------步骤二：用户名以及邮箱填充【将汉语转成拼音填充】【"+channelHandlerRequest.getRequestId()+"】");
        Object params = channelHandlerRequest.getParams();
        if(params instanceof User){
            User user = (User)params;
            String fullname = user.getFullname();
            HanyuPinyinOutputFormat hanyuPinyinOutputFormat = new HanyuPinyinOutputFormat();
            hanyuPinyinOutputFormat.setToneType(HanyuPinyinToneType.WITHOUT_TONE);
            String username = PinyinHelper.toHanYuPinyinString(fullname, hanyuPinyinOutputFormat);
            user.setUsername(username);
            user.setEmail(username + "@qq.com");
            return true;
        }


        return false;
    }
}

 public class UserPwdEncryptChannelHandler extends AbstactChannelHandler {
    @Override
    public boolean handler(ChannelHandlerContext chx) {
        ChannelHandlerRequest channelHandlerRequest = chx.getChannelHandlerRequest();
        System.out.println("------------------------------------步骤三：用户密码明文转密文【"+channelHandlerRequest.getRequestId()+"】");
        Object params = channelHandlerRequest.getParams();
        if(params instanceof User){
            String encryptPwd = DigestUtil.sha256Hex(((User) params).getPassword());
            ((User) params).setPassword(encryptPwd);
            return true;
        }

        return false;
    }
}

 public class UserMockSaveChannelHandler extends AbstactChannelHandler {

    @Override
    public boolean handler(ChannelHandlerContext chx) {
        ChannelHandlerRequest channelHandlerRequest = chx.getChannelHandlerRequest();
        System.out.println("------------------------------------步骤四：模拟用户数据落库【"+channelHandlerRequest.getRequestId()+"】");
        Object params = channelHandlerRequest.getParams();
        if(params instanceof User){
            Map<String, User> userMap = new HashMap<>();
            User user = (User)params;
            userMap.put(user.getUsername(),user);
            chx.put("userMap",userMap);
            return true;
        }


        return false;
    }
}

 public class UserPrintChannleHandler extends AbstactChannelHandler {
    @Override
    public boolean handler(ChannelHandlerContext chx) {
        ChannelHandlerRequest channelHandlerRequest = chx.getChannelHandlerRequest();
        System.out.println("------------------------------------步骤五：打印用户数据【"+channelHandlerRequest.getRequestId()+"】");
        Object params = channelHandlerRequest.getParams();
        if(params instanceof User){
            Object userMap = chx.get("userMap");
            if(userMap instanceof Map){
                Map map = (Map)userMap;
                if(map.containsKey(((User) params).getUsername())){
                    System.out.println(map.get(((User) params).getUsername()));
                    return true;
                }
            }
        }

        return false;
    }
}

5. Arrange and combine each sub-task

 @Service
public class UserServiceImpl implements UserService {

    @Override
    public boolean save(User user) {
       return ChannelPipelineExecutor.pipeline()
                .addLast(new UserCheckChannelHandler())
                .addLast(new UserFillUsernameAndEmailChannelHandler())
                .addLast(new UserPwdEncryptChannelHandler())
                .addLast(new UserMockSaveChannelHandler())
                .addLast(new UserPrintChannleHandler())
                .start(ChannelHandlerRequest.builder().params(user).build());
    }
}

6. Test

 Faker faker = Faker.instance(Locale.CHINA);
        User user = User.builder().age(20)
                .fullname(faker.name().fullName())
                .mobile(faker.phoneNumber().phoneNumber())
                .password("123456").build();
        userService.save(user);

View the console

Think about it: Is there any room for optimization in the pipeline mode implemented above?

In step 5, each sub-task is arranged and combined. Assuming that there are N steps in the sub-service, we need to addLast N times, which feels a bit hard-coded. So we can do the following transformation

Retrofit

1. Define pipeline annotations

 @Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Component
@Role(BeanDefinition.ROLE_INFRASTRUCTURE)
public @interface Pipeline {

    Class consumePipelinesService();

    String consumePipelinesMethod();

    Class[] args() default {};

    int order();
}

2. Define the pipeline scanner

 public class PipelineClassPathBeanDefinitionScanner extends ClassPathBeanDefinitionScanner {

    public PipelineClassPathBeanDefinitionScanner(BeanDefinitionRegistry registry) {
        super(registry);
    }


    @Override
    protected Set<BeanDefinitionHolder> doScan(String... basePackages) {
        Set<BeanDefinitionHolder> beanDefinitionHolders = super.doScan(basePackages);
        for (BeanDefinitionHolder beanDefinitionHolder : beanDefinitionHolders) {
            GenericBeanDefinition beanDefinition = (GenericBeanDefinition) beanDefinitionHolder.getBeanDefinition();
            String className = beanDefinition.getBeanClassName();
            beanDefinition.getPropertyValues().addPropertyValue("pipelineServiceClz",className);
            beanDefinition.setBeanClass(ComsumePipelineFactoryBean.class);

        }

        return beanDefinitionHolders;

    }

    @Override
    protected boolean isCandidateComponent(AnnotatedBeanDefinition beanDefinition) {
        return beanDefinition.getMetadata().isInterface();
    }
}

3. Define the pipeline registrar

 public class PipelineImportBeanDefinitionRegistrar implements ImportBeanDefinitionRegistrar {

    @Override
    public void registerBeanDefinitions(AnnotationMetadata importingClassMetadata, BeanDefinitionRegistry registry) {


        PipelineClassPathBeanDefinitionScanner scanner = new PipelineClassPathBeanDefinitionScanner(registry);
        scanner.addIncludeFilter(new AnnotationTypeFilter(FunctionalInterface.class));
        Set<String> basePackages = getBasePackages(importingClassMetadata);
        String[] basePackageArr = {};
        scanner.scan(basePackages.toArray(basePackageArr));

    }

    protected Set<String> getBasePackages(AnnotationMetadata importingClassMetadata) {
        Map<String, Object> attributes = importingClassMetadata.getAnnotationAttributes(EnabledPipeline.class.getCanonicalName());

        Set<String> basePackages = new HashSet<>();

        for (String pkg : (String[]) attributes.get("basePackages")) {
            if (StringUtils.hasText(pkg)) {
                basePackages.add(pkg);
            }
        }

        if (basePackages.isEmpty()) {
            basePackages.add(
                    ClassUtils.getPackageName(importingClassMetadata.getClassName()));
        }
        return basePackages;
    }
}

4. Define the EnableXXX annotation

 @Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@Documented
@Import(PipelineImportBeanDefinitionRegistrar.class)
public @interface EnabledPipeline {

    String[] basePackages() default {};
}

Note: In addition, the pipeline proxy and pipeline factoryBean need to be defined, because the space will not be posted. Interested friends can check the demo link at the end of the article

5. Transform the original pipeline task executor into the following

 @Slf4j
@Pipeline(consumePipelinesService = UserService.class,consumePipelinesMethod = "save",args = {User.class},order = 1)
public class UserCheckChannelHandler extends AbstactChannelHandler {

    
    @Override
    public boolean handler(ChannelHandlerContext chx) {
        ChannelHandlerRequest channelHandlerRequest = chx.getChannelHandlerRequest();
        System.out.println("------------------------------------步骤一：用户数据校验【"+channelHandlerRequest.getRequestId()+"】");
        String json = JSON.toJSONString(channelHandlerRequest.getParams());
        List<User> users = JSON.parseArray(json,User.class);
        if(CollectionUtil.isEmpty(users) || StringUtils.isBlank(users.get(0).getFullname())){
            log.error("用户名不能为空");
            return false;
        }
        return true;


    }
}

 @Slf4j
@Pipeline(consumePipelinesService = UserService.class,consumePipelinesMethod = "save",args = {User.class},order = 2)
public class UserFillUsernameAndEmailChannelHandler extends AbstactChannelHandler {
    @SneakyThrows
    @Override
    public boolean handler(ChannelHandlerContext chx) {
        ChannelHandlerRequest channelHandlerRequest = chx.getChannelHandlerRequest();
        System.out.println("------------------------------------步骤二：用户名以及邮箱填充【将汉语转成拼音填充】【"+channelHandlerRequest.getRequestId()+"】");
        String json = JSON.toJSONString(channelHandlerRequest.getParams());
        List<User> users = JSON.parseArray(json,User.class);
        if(CollectionUtil.isNotEmpty(users)){
            User user = users.get(0);
            String fullname = user.getFullname();
            HanyuPinyinOutputFormat hanyuPinyinOutputFormat = new HanyuPinyinOutputFormat();
            hanyuPinyinOutputFormat.setToneType(HanyuPinyinToneType.WITHOUT_TONE);
            String username = PinyinHelper.toHanYuPinyinString(fullname, hanyuPinyinOutputFormat);
            user.setUsername(username);
            user.setEmail(username + "@qq.com");
            return true;

        }



        return false;
    }
}

. . . Omit remaining pipeline task executors

6. The original step arrangement, only need to write the interface

 @FunctionalInterface
public interface UserService {

    boolean save(User user);

}

This is all it takes to orchestrate

7. Test

Add the @EnabledPipeline annotation to the startup class. Examples are as follows

 @SpringBootApplication
@EnabledPipeline(basePackages = "com.github.lybgeek.pipeline.spring.test")
public class SpringPipelineApplication  {

    public static void main(String[] args) {
        SpringApplication.run(SpringPipelineApplication.class);
    }

}

 @Test
    public void testPipeline(){
        boolean isOk = userService.save(user);
        Assert.assertTrue(isOk);

    }

The effect of the arrangement is the same as before

Summarize

This paper mainly implements two different forms of pipeline modes. One is based on annotations. The orchestration steps are directly written on the executor through annotations, and the executor is used to locate the business execution method. The other is to combine and call the executor by yourself in the business method. Although this method avoids the business method to arrange the executors by themselves, there is also a need to turn over each executor class to see the order of its executors when there are too many executors. , and can not achieve the combination effect we want. Based on this question, I will introduce two other implementation methods in the next article.

demo link

https://github.com/lyb-geek/springboot-learning/tree/master/springboot-pipeline