Use Elastic Job's sharding configuration to accelerate task execution and improve resource utilization

In the previous article, we introduced how to use Elastic Job to implement timing tasks . It solves the @Scheduled to realize it, and also realizes the high-availability execution of timing tasks.

However, there is another type of problem that is easy to occur when we do timed tasks, that is, the task execution speed is too long; at the same time, in order to achieve high availability of timed tasks, many task instances have been started, but each task is executed. The instance is running, and the resource utilization is not high.

So, let's continue to introduce the use of Elastic Job's sharding configuration to accelerate task execution and raise the goal of resource utilization!

Try it

chapter7-2 project in the warehouse at the end of the article, and then modify it on this basis. Of course, if you haven't entered how to use Elastic Job, then go to the previous article to be a knowledge base, and then continue the following content!

first step : Create a task to be executed by shards

@Slf4j
@Service
public class MyShardingJob implements SimpleJob {

    @Override
    public void execute(ShardingContext context) {
        switch (context.getShardingItem()) {
            case 0:
                log.info("分片1：执行任务");
                break;
            case 1:
                log.info("分片2：执行任务");
                break;
            case 2:
                log.info("分片3：执行任务");
                break;
        }
    }

}

Here, switch used to determine the sharding-item value of the current task context to execute different sharding tasks. The value of sharding-item depends on the total number of shards to be configured later, but note that it starts counting from 0. Here only the log printing method is used to show the effect of fragmentation. When realizing the business logic, you must remember to design the fragmentation operation for the execution task according to the number of fragments. For example, you can distinguish between different shards and process different data according to the way of calculating the ID of the batch task, so as to avoid problems caused by repeated execution.

Second step : In the configuration file, set the implementation class of the configuration task, the execution expression, and the parameters of the total number of shards to be tested.

elasticjob.jobs.my-sharding-job.elastic-job-class=com.didispace.chapter73.MyShardingJob
elasticjob.jobs.my-sharding-job.cron=0/5 * * * * ?
elasticjob.jobs.my-sharding-job.sharding-total-count=3

It is set to 3 here, so the task will be divided into 3 shards, and each shard corresponds to a switch branch in the first step.

Run and test

Single instance operation

After completing the above code, try to start the first instance of the above implementation.

At this point, we can see that every 5 seconds, this instance will print a log like this:

2021-07-21 17:42:00.122  INFO 63478 --- [           main] .s.b.j.ScheduleJobBootstrapStartupRunner : Starting ElasticJob Bootstrap.
2021-07-21 17:42:00.126  INFO 63478 --- [           main] org.quartz.core.QuartzScheduler          : Scheduler my-sharding-job_$_NON_CLUSTERED started.
2021-07-21 17:42:00.126  INFO 63478 --- [           main] .s.b.j.ScheduleJobBootstrapStartupRunner : ElasticJob Bootstrap started.
2021-07-21 17:42:05.254  INFO 63478 --- [-sharding-job-1] com.didispace.chapter73.MyShardingJob    : 分片1：执行任务
2021-07-21 17:42:05.254  INFO 63478 --- [-sharding-job-3] com.didispace.chapter73.MyShardingJob    : 分片3：执行任务
2021-07-21 17:42:05.254  INFO 63478 --- [-sharding-job-2] com.didispace.chapter73.MyShardingJob    : 分片2：执行任务
2021-07-21 17:42:10.011  INFO 63478 --- [-sharding-job-4] com.didispace.chapter73.MyShardingJob    : 分片1：执行任务
2021-07-21 17:42:10.011  INFO 63478 --- [-sharding-job-5] com.didispace.chapter73.MyShardingJob    : 分片2：执行任务
2021-07-21 17:42:10.011  INFO 63478 --- [-sharding-job-6] com.didispace.chapter73.MyShardingJob    : 分片3：执行任务

Each task is split into 3 shard tasks, as I said above, each shard corresponds to a switch branch. In the current situation, we only start one instance, so the three sharding tasks are all assigned to this unique instance.

Dual instance operation

Next, we start another instance (note the use of -Dserver.port to change a different port, otherwise the local startup will fail). At this time, the logs of the two instances have changed:

Example 1 log:

2021-07-21 17:44:50.190  INFO 63478 --- [ng-job_Worker-1] com.didispace.chapter73.MyShardingJob    : 分片2：执行任务
2021-07-21 17:44:55.007  INFO 63478 --- [ng-job_Worker-1] com.didispace.chapter73.MyShardingJob    : 分片2：执行任务
2021-07-21 17:45:00.010  INFO 63478 --- [ng-job_Worker-1] com.didispace.chapter73.MyShardingJob    : 分片2：执行任务

Example 2 log:

2021-07-21 17:44:50.272  INFO 63484 --- [-sharding-job-1] com.didispace.chapter73.MyShardingJob    : 分片1：执行任务
2021-07-21 17:44:50.273  INFO 63484 --- [-sharding-job-2] com.didispace.chapter73.MyShardingJob    : 分片3：执行任务
2021-07-21 17:44:55.009  INFO 63484 --- [-sharding-job-3] com.didispace.chapter73.MyShardingJob    : 分片1：执行任务
2021-07-21 17:44:55.009  INFO 63484 --- [-sharding-job-4] com.didispace.chapter73.MyShardingJob    : 分片3：执行任务

As the number of instances increases, you can see that the allocation of shards has changed. This also means that when a task starts to execute, both task execution instances are used, so that the efficiency of our task execution and resource utilization can be optimized.

You can also try to continue to start the instance and close the instance to observe the dynamic allocation of tasks, how about it? Isn't it easier to write timed tasks like this? Be sure to remember to write by yourself, so that you will have a deeper experience! If you encounter a problem, you can pull the code example at the end of the article to compare whether there is a difference in configuration. In the next article, we will continue to introduce some advanced content about timed tasks.

This series of tutorials "Spring Boot 2.x basic tutorial" click directly! , welcome to collect and forward! What if you encounter difficulties in the learning process? You can join our Spring technical exchange group , participate in exchanges and discussions, and learn and progress better!

Code example

The complete project of this article can be viewed in the chapter7-3 directory in the following warehouse:

• Github：https://github.com/dyc87112/SpringBoot-Learning/
• Gitee：https://gitee.com/didispace/SpringBoot-Learning/

If you think this article is good, welcome Star support, your attention is my motivation for persistence!

Welcome to pay attention to my public account: Program Ape DD, share knowledge and thoughts that can’t be seen elsewhere