Serverless JOB | New changes in traditional tasks

As a type of load that can be stopped after running, Job has rich usage scenarios in enterprise-level development. SAE Job extends the inclusive dividends brought by serverless technology from the application field to the task field. By combining the usage scenarios and best practices of longrun + shortrun, it is built into a serverless integrated enterprise-level development and operation platform to meet the needs of different industries. To provide users with more complete and diverse capability support and stable and reliable technical support.

At present, the mainstream way to run Job is to use distributed task framework, such as Quartz, XXLJob, ElasticJob, etc. As an open source project that has been around for a long time, such frameworks are used by many enterprises and have mature functions. In the cloud-native era, K8S Job and CronJob are gradually being considered for adoption. However, the above solutions generally have the following pain points:

First, resource utilization is low. The use of an open source distributed framework requires the program to be resident, and it is charged for 7*24 hours in the cloud host. The K8S solution also requires users to maintain cluster nodes, resulting in a waste of costs.

Second, the observability is poor. Users need to build a complete set of log collection, cluster and business monitoring indicator collection, and alarm systems to meet the needs of the production environment.

Finally, operations and maintenance are complex. Whether it is an open source framework or K8S, it is necessary to pay attention to the high availability of the underlying resources, the capacity and elasticity of tasks under high concurrency, and its operation and maintenance operations have high technical complexity.

As the first task-oriented Serverless PaaS platform, SAE Job perfectly solves the above pain points. SAE Job focuses on solving the user's efficiency and cost issues. It combines the experience and functions of traditional tasks while using it on demand, billing by volume, enabling low-threshold tasks to go to the cloud and saving the cost of idle resources. At the same time, the experience adopts event-driven and non-intrusive task scheduling and control, and users can have out-of-the-box functions such as full life cycle management and observability of tasks with zero modification.

SAE Job supports a variety of calling methods, including Alibaba Cloud standard API/SDK, which can implement scheduled tasks through visual configuration of Cron expressions, and pull up SAE Jobs through various triggers such as HTTP/MQ/OSS. At the same time, it supports many core features of tasks, including task life cycle management, execution record, event notification, log monitoring and alarm, timeout retry, blocking strategy, task sharding, task multi-concurrency, etc.

SAE Job provides three core values:

① Complete and fully managed: It provides a one-stop fully managed management interface. Its task life cycle management and observability are out of the box. Users can learn to use SAE with low mental burden and zero cost.
② Simple operation and maintenance free: The underlying resources are shielded, and users only need to focus on the development of their core business logic, without worrying about cluster availability, capacity, performance and other issues.
③ Super cost-effective: Adopt the mode of on-demand use and pay-as-you-go. Only when the task executes business logic will it be charged, and no fee will be charged for the rest of the time, which greatly saves the cost of resources.
The following demonstrates the overall usage process of SAE Job:

Jingxiao demo 01.mp4

SAE Job is task-centric and provides a traditional user experience. Currently, it focuses on tasks that support single-machine broadcast and parallel sharding models, and supports event-driven, concurrency strategies, and timeout retry, and many other features. It provides low-cost, multi-specification, and highly elastic resource instances to meet the execution of short-term tasks.

Compared with open-source distributed frameworks, its advantages lie in the user experience of full-hosted operation and maintenance, complete out-of-the-box functions, and white-screen management and control. After tasks are run, resources are released immediately without wasting idle resource costs.

Compared with K8S Job, its advantages are not only free of full hosting, but also users do not need to understand K8S related concepts and technical details, and do not need to maintain its complexity.

SAE supports the transformation and migration of XXL Job 0. Users can deploy XXL JOB applications to SAE Jobs without any code and configuration modifications, and users only need to pay for the actual execution logic of the task. In this process, SAE Job acts as the scheduling center and executor of XXL Job. Users only need to focus on task code and simple configuration, such as task templates, concurrent retry, etc., and SAE is responsible for task scheduling and control without intrusion.

The core value of deploying XXL Job to SAE is cost reduction and efficiency improvement:

Descending the body now: If the original XXL job is used, in order to maintain its high availability, at least the resident cost of MySQL+2ECS+SLB+N*ECS is required, while deploying to SAE only needs to perform specific business logic for its task. The consumption of CPU memory is paid, that is, the actual resource consumption of SAE.

The efficiency improvement is reflected in: the SAE fully managed plane operation and maintenance experience reduces the overall operation and maintenance complexity and improves application availability.

The following demonstrates the XXL Job 0 transformation and migration process:
Jingxiao demo 02.mp4

SAE Job currently focuses on pan-Internet, new retail, e-commerce, cultural media, manufacturing, IoT, logistics, financial securities, healthcare and insurance industries. The main scenes include the following six:

① Scheduled tasks: regularly pull data and crawlers.
② Batch processing: data cleaning, conversion, analysis.
③ Asynchronous execution: Status refresh and offline query are performed asynchronously.
④ Traditional framework migration: XXL Job 0 transformation and migration, etc.
⑤ Microservice architecture: call communication and process decoupling with the original microservice architecture.
⑥ CI/CD: Implement GitOps with SAE Job as a carrier for building images, thereby improving the CI/CD process.

Finally, I will share a customer case that uses SAE Job combined with microservices. The user's business appeal is: It is necessary to regularly push the data of hotel product changes to third-party platforms, such as Fliggy, through scheduled tasks. Its task has two characteristics: first, the initialization of the task takes a long time and takes minutes; second, the task execution time is very long, taking 5-6 hours, and in addition to processing business logic, it also needs to call other microservices to Get information such as metadata.

The solution we provide for this is to directly deploy microservices such as order center, product center, and user center to SAE applications, and deploy scheduled tasks to SAE Jobs. Users can communicate through SAE's built-in registration center without modification.

The solution provides users with many values:

• The two types of loads have unified entry operations and call 0 transformation between applications.
• The task is stopped immediately after the task is completed, and the idle resources are released immediately, which greatly saves the resource cost.
• Automatically retry if timeout fails, and realize self-healing without manual intervention.
• Provides a complete task runtime monitoring and alarm mechanism.

With this solution, the user 0 code transformation completes the serverless architecture of the entire architecture, while saving resource costs and operation and maintenance costs. SAE will continue to escort the availability of its applications and tasks.

Finally, everyone is welcome to use SAE Job, the first task-oriented Serverless PaaS platform, to experience the new changes it brings to traditional tasks.

For more content, pay attention to the Serverless WeChat official account (ID: serverlessdevs), which brings together the most comprehensive content of serverless technology, regularly holds serverless events, live broadcasts, and user best practices.