A full review of mainstream timed task solutions

Author: Jing Xiao

As a general mode of executing expected logic according to the agreed time, scheduled tasks carry rich business scenarios in enterprise-level development, such as background synchronization of data to generate reports, regular cleaning of disk log files, and regular scanning of overtime orders for compensation callbacks. Program developers have many frameworks and solutions to choose from in the field of scheduled tasks, and use this to quickly realize business functions and achieve product launch. This article will introduce and analyze the current mainstream timed task solutions, and hope that it can be used as a reference in enterprise technology selection and project architecture reconstruction.

Crontab

target setting

As a built-in executable command in Linux, Crontab can execute the specified system command or shell script according to the time generated by the cron expression.

How to use

crontab command syntax:

 crontab [-u username] [-l | -e | -r ]
参数：
-u : 只有root用户才能进行这个任务，编辑某个用户的crontab
-e : 编辑 crontab 的工作内容
-l : 查阅 crontab 的工作内容
-r : 移除所有的 crontab 的工作内容

Configuration file example:

 * * * * * touch ~/crontab_test
* 3 * * * ~/backup
0 */2 * * * /sbin/service httpd restart

Implementation principle

The crond daemon is started through the init process when Linux starts. Cornd checks the /etc/crontab configuration file every minute for tasks that need to be executed, and outputs the execution of the scheduled tasks through the /var/log/cron file. Users can manage the /etc/crontab configuration file using the Crontab command.

case analysis

With Crontab, users can quickly and easily implement simple scheduled tasks, but there are the following pain points:

• The scheduled task is bound to the specified linux machine. When the machine is expanded or replaced, the contab needs to be reconfigured, and there is a risk of a single point of failure.
• As the scale of scheduled tasks increases, there is no unified perspective to track and control the task progress, which is difficult to maintain.
• The function is too simple, there is no advanced task features such as timeout, retry, blocking and so on
• Poor observability, difficult to troubleshoot and locate
• Tasks are resident, causing unnecessary waste of resource costs when no tasks are executed

Spring Task

target setting

The Spring framework provides out-of-the-box timing scheduling functions. Users can identify the execution cycle of a specified method by xml or @Scheduled annotation. Spring Task supports multiple task execution modes, including corn with time zone configuration, fixed delay, fixed rate, etc.

How to use

The code example is as follows:

 @EnableScheduling
@SpringBootApplication
public class App {

    public static void main(String[] args) {
        SpringApplication.run(App.class, args);
    }
}

@Component
public class MyTask {  

    @Scheduled(cron = "0 0 1 * * *")
    public void test() {
        System.out.println("test");  
    }  

}

Implementation principle

The principle of Spring Task is to intercept the method identified by the @Scheduled annotation with the help of ScheduledAnnotationBeanPostProcessor when initializing the bean, and build the corresponding Task instance according to each method and its annotation configuration and register it in the ScheduledTaskRegistrar, and call back through afterSingletonsInstantiated after the initialization of the singleton bean is completed. Set the scheduler TaskScheduler in ScheduledTaskRegistrar, whose bottom layer depends on the implementation of ScheduledThreadPoolExecutor in jdk concurrent package, and add all Tasks to TaskScheduler for scheduling execution in afterPropertiesSet.

case analysis

With Spring Task, users can quickly implement periodic execution of specified methods through annotations, and support multiple periodic strategies. But similar to crontab, it also has the following pain points:

• The default is single-threaded execution. If the execution time of the previous task is long, the subsequent tasks will be blocked by starvation. Users need to configure the thread pool by themselves.
• Each node runs independently, there is a single point of risk, there is no distributed coordination mechanism, and concurrent execution should be considered prohibited
• As the scale of scheduled tasks increases, there is no unified perspective to track and control the task progress, which is difficult to maintain.
• The function is too simple, there is no advanced task features such as timeout, retry, blocking and so on
• Poor observability, difficult to troubleshoot and locate
• Tasks are resident, causing unnecessary waste of resource costs when no tasks are executed

ElasticJob

target setting

As an open source distributed task framework of Dangdang.com, ElasticJob provides many features such as elastic scheduling, resource management and control, and job governance. It has become a sub-project of Apache Shardingsphere. ElasticJob currently consists of two independent sub-projects, ElasticJob-Lite and ElasticJob-Cloud. ElasticJob-Lite is positioned as a lightweight and decentralized solution, providing coordination services for distributed tasks in the form of jars; ElasticJob-Cloud uses Mesos The solution provides additional services such as resource governance, application distribution, and process isolation. Generally, ElasticJob-Lite can be used to meet the needs.

How to use

The user needs to configure the address of the registry zk and the configuration information of the task in yaml:

 elasticjob:
  regCenter:
    serverLists: localhost:6181
    namespace: elasticjob-lite-springboot
  jobs:
    simpleJob:
      elasticJobClass: org.apache.shardingsphere.elasticjob.lite.example.job.SpringBootSimpleJob
      cron: 0/5 * * * * ?
      timeZone: GMT+08:00
      shardingTotalCount: 3
      shardingItemParameters: 0=Beijing,1=Shanghai,2=Guangzhou

The corresponding task can be identified by implementing the corresponding interface, and the callback before and after task execution can be realized by configuring the listener:

 public class MyElasticJob implements SimpleJob {

    @Override
    public void execute(ShardingContext context) {
        switch (context.getShardingItem()) {
            case 0: 
                // do something by sharding item 0
                break;
            case 1: 
                // do something by sharding item 1
                break;
            case 2: 
                // do something by sharding item 2
                break;
            // case n: ...
        }
    }
}


public class MyJobListener implements ElasticJobListener {

    @Override
    public void beforeJobExecuted(ShardingContexts shardingContexts) {
        // do something ...
    }

    @Override
    public void afterJobExecuted(ShardingContexts shardingContexts) {
        // do something ...
    }

    @Override
    public String getType() {
        return "simpleJobListener";
    }
}

Implementation principle

The underlying time scheduling of ElasticJob is based on Quartz. Quartz needs to persist business beans to the underlying data table. The system is quite intrusive. At the same time, task preemption is performed through db locks, which does not support parallel scheduling and is not scalable. ElasticJob achieves horizontal expansion through the characteristics of data sharding and custom sharding parameters. It can split a task into N independent task items, and the distributed servers execute the assigned shard items in parallel. For example, if there are 100 million pieces of data in a database, and these data need to be read out and calculated, the 100 million pieces of data can be divided into 10 shards, and each shard reads 10 million pieces of data, Then write to database after calculation. If there are three machines for execution, machine A is divided into shards (0, 1, 2, 9), machine B is divided into shards (3, 4, 5), and machine C is divided into shards (6, 7, 8), This is also the most significant advantage over Quartz.

Realize that ElasticJob uses zookeeper as the registry for distributed scheduling coordination and leader node election, and senses the increase or decrease of servers through the changes of temporary nodes in the registry. Whenever the leader node is elected, the server goes offline, and the total number of shards changes, the Subsequent resharding will be triggered, and the leader node will perform specific sharding when the next scheduled task is triggered, and then each node will obtain the sharding information from the registry and execute it as the task's running parameters.

case analysis

As a distributed task framework, ElasticJob solves the problem that the above single-node tasks cannot guarantee high availability and high concurrency performance during task execution, and supports failover and missed job re-execution (misfire), etc. Advanced mechanism, but there are still the following pain points in the use process:

• The overall framework is heavy and needs to rely on the external registration center zk, which increases the use cost and maintenance complexity of at least three servers
• As the amount of tasks continues to increase, zk as a stateful middleware will become a performance bottleneck
• Weak observability, need to introduce additional db and configure event tracking
• Tasks are resident, causing unnecessary waste of resource costs when no tasks are executed

XXLJob

target setting

XXLJob is a distributed task framework open sourced by Dianping employees. Its core design goals are rapid development, easy learning, light weight, and easy expansion. XXLJob has rich functions, such as task segment broadcasting, timeout control, failure retry, blocking strategy, etc., and manages and maintains tasks through an experience-friendly white-screen console.

How to use

XXLJob is divided into two parts: a central scheduler and a distributed executor, which need to be started separately when used, and the dependent db configuration needs to be configured when the scheduling center starts.

The executor needs to configure the address of the dispatch center:

 xxl.job.admin.addresses=http://127.0.0.1:8080/xxl-job-admin
xxl.job.accessToken=
xxl.job.executor.appname=xxl-job-executor-sample
xxl.job.executor.address=
xxl.job.executor.ip=
xxl.job.executor.port=9999
xxl.job.executor.logpath=/data/applogs/xxl-job/jobhandler
xxl.job.executor.logretentiondays=30

Tasks created in the form of a bean pattern method and used before and after callbacks are as follows:

 @XxlJob(value = "demoJobHandler2", init = "init", destroy = "destroy")
public void demoJobHandler() throws Exception {
   int shardIndex = XxlJobHelper.getShardIndex();
   int shardTotal = XxlJobHelper.getShardTotal();
   XxlJobHelper.log("分片参数：当前分片序号 = {}, 总分片数 = {}", shardIndex, shardTotal);
}

public void init(){
        logger.info("init");
}

public void destroy(){
        logger.info("destory");
}

After the task is created, you need to configure the task execution policy on the console:

Implementation principle

The implementation of XXLJob decouples the scheduling system from tasks. Its self-developed scheduler is responsible for managing scheduling information, issuing scheduling requests according to the scheduling configuration, supporting visual, simple and dynamic management of scheduling information, automatic discovery and routing, scheduling expiration policies, and retrying Strategies that support the executor Failover. The executor is responsible for receiving scheduling requests and executing task logic, as well as receiving task termination requests and log requests, and is responsible for task timeouts, blocking policies, etc. The scheduler and the executor communicate through restful api. The scheduler itself supports cluster deployment stateless, improves the disaster tolerance and availability of the scheduling system, and maintains lock information and persistence through mysql. The executor stateless supports cluster deployment, improves the availability of the scheduling system, and improves the task processing capability.

A complete task scheduling communication process of XXLJob: First, the scheduling center sends an http scheduling request to the executor embedded server, and then the executor executes the corresponding task logic. After the task execution is completed or the executor times out, the executor sends an http callback to the scheduling center to return the scheduling result .

case analysis

XXLJob is widely popular in the open source community. With its simple operation and rich functions, it has been put into use in many companies, which can better meet the needs of production level, but the following pain points need to be improved:

• It needs to rely on an external DB, which increases the use cost and maintenance complexity of the database
• Relying on DB locks to ensure the consistency of cluster distributed scheduling, when the amount of short-term tasks continues to increase, it will put greater pressure on DB and become a performance bottleneck
• Compared with the centerless mode, which requires additional deployment of schedulers, both schedulers and executors need to be resident and at least two are required to ensure high availability. When no tasks are executed, unnecessary resource costs are wasted.

Serverless Job

target setting

As the best practice and future evolution trend of cloud computing, serverless is highly respected in the cloud-native era due to its fully managed and O&M-free experience and the cost advantage of pay-as-you-go. SAE (Serverless Application Engine) Job, as the first task-oriented Serverless PaaS platform, provides traditional user experience. The current focus is on tasks that support single-machine, broadcast, and parallel sharding models. It also supports event-driven, concurrency policies, and timeout retry and many other features. It provides low-cost, multi-specification, and highly elastic resource instances to meet the execution of short-term tasks.

How to use

For existing applications using all the above solutions, SAE (Serverless Application Engine) Job supports zero-transformation and non-inductive migration while being compatible with functional experience. Whether users are using Crontab, Spring Task, or ElasticJob, XXLJob, the code package or XXLJob can be used. The image is directly deployed into the SAE (Serverless Application Engine) job, and directly upgraded to a serverless architecture, thus immediately having the technical advantages brought by serverless, saving resource costs and operation and maintenance costs.

For programs that stop after running, whether it is Java, Shell, Python, Go, or Php, they can be directly deployed to SAE (Serverless Application Engine) Jobs, so that you can immediately have a white screen control and a complete experience of full hosting and free operation and maintenance. And out-of-the-box observables.

Implementation principle

The bottom layer of SAE (Serverless Application Engine) Job is multi-tenant Kubernetes, which uses Shenlong bare metal security container and VK to connect to ECI to provide cluster computing resources. Tasks run by users in SAE (Serverless Application Engine) are mapped to corresponding resources in Kubernetes. The multi-tenancy capability is to realize the isolation between tenants by means of system isolation, data isolation, service isolation and network isolation. SAE (Serverless Application Engine) Job uses Event Bridge as an event-driven source, which not only supports rich calling methods, but also avoids the problems of Kubernetes built-in timers that do not guarantee punctual triggering and precision time zones. At the same time, the enterprise-level features of the Job controller are continuously improved, and mechanisms such as custom sharding, injection configuration, differentiated GC, active sidecar exit, real-time log persistence, and event notification are added. And with the help of Java bytecode enhancement technology, it takes over task scheduling to realize the zero-transformation serverless of the general Java target framework. Using the KubeVela software delivery platform as the carrier of task release and management, the entire cloud-native delivery is completed in a declarative manner with task-centric, open source and open standards. SAE (Serverless Application Engine) Job will continue to optimize the efficiency of etcd and the scheduler in the scenario of short-term task and high-concurrency creation, as well as the ultimate elasticity of instance startup, combined with elastic resource pools to ensure low latency and high availability of task scheduling.

case analysis

SAE (Serverless Application Engine) Job solves the various pain points of the above-mentioned timed task solutions. Users do not need to pay attention to task scheduling and cluster resources, do not need to deploy additional operation and maintenance dependent components, and do not need to build a complete set of monitoring and alarming systems. More importantly, there is no need for 7*24h resident cloud hosts, and idle resources are continuously consumed in an environment with low resource utilization.

Compared with traditional scheduled task solutions, SAE (Serverless Application Engine) Job provides three core values:

• Complete and fully managed: It provides a one-stop fully managed management interface. Its task lifecycle management and observability are out of the box. Users can learn to use SAE (Serverless Application Engine) with low mental burden and zero cost.
• Simple operation and maintenance free: The underlying resources are shielded, and users only need to focus on the development of their core business logic, without worrying about cluster availability, capacity, performance and other issues.
• Super cost-effective: Adopt the mode of on-demand use and pay-as-you-go. Only when the task executes business logic will it be charged, and no fee will be charged for the rest of the time, which greatly saves the cost of resources.

Summarize

This article expounds the target positioning, usage, and implementation principles of mainstream timed task solutions (Crontab, Spring Task, ElasticJob, XXLJob, Serverless Job), and at the same time, makes a horizontal evaluation on the functional experience and performance costs that enterprises pay close attention to. analyze. Finally, I hope that everyone will choose Serverless Job and experience the new changes it brings to traditional tasks.

---

Serverless Developer Meetup

2022 Alibaba Cloud Serverless Developer Meetup Hangzhou Station is hotly registered!

This salon specially invites serverless front-line technical experts from Alibaba Cloud and AutoNavi to share the latest exploration and research on the current serverless architecture CICD; bring the cost reduction and efficiency increase practice of serverless in AutoNavi business system; analyze the atomization of serverless from a comprehensive perspective Ability is how to improve R&D efficiency and enhance development happiness. There is also a Serverless Workshop waiting for you to play, hand-in-hand with you to experience the fun of 1-minute rapid deployment, and feel the beauty of Severless computing power. Sign up for free to participate in the event, we have prepared a lot of peripheral gifts, waiting for you to come to Serverless!

• Time: September 17 (Saturday) 13:00
• Venue: Visitor Center, Area B, Alibaba Xixi Park

Click here to register now!