5 ways Python uses scheduled tasks

Abstract: Python has several ways to schedule a task regularly, this is what we will learn in this article.

This article is shared from Huawei Cloud Community " to Use Schedule Jobs in Python) , author: Regan Yue.

Most applications built today require some sort of scheduling mechanism. Polling APIs or databases, constantly checking system health, archiving logs, etc. are common examples. Software that uses Auto-scaling, such as Kubernetes and Apache Mesos, needs to check the status of deployed applications. For this reason, they use liveness probes that run regularly. Scheduling tasks need to be decoupled from business logic, so we need to use decoupled execution queues, such as Redis queues.

There are several ways to schedule a task in Python. This is what we will learn in this article. I will discuss scheduling tasks in the following way:

1. Simple Loops
2. Simple Loops but Threaded
3. Schedule Library
4. Python Crontab
5. RQ Scheduler as decoupled queues

Simple loops

It is effortless to use simple loops to implement scheduling tasks. Using an infinitely running while loop to periodically call a function can be used to schedule jobs, but this is not the best method, but it is very effective. You can use the slleep() of the built-in time module to delay execution. But this is not the way most jobs are scheduled, because it looks ugly and it is less readable compared to other methods.

import time
​
def task():
    print("Job Completed!")
​
 while 1:
    task()
    time.sleep(10)

When it comes to schedules such as 9:00 a.m. or 7:45 p.m. every Wednesday, things get trickier.

import datetime
​
def task():
    print("Job Completed!")
​
 while 1:
    now = datetime.datetime.now()
    # schedule at every wednesday,7:45 pm
    if now.weekday == 3 and now.strftime("%H:%m") == "19:45":
        task()
    # sleep for 6 days
    time.sleep(6 * 24 * 60 * 60)

This is the solution I thought of for the first time, thanks! One problem with this method is that the logic here is blocking, that is, once this code is found in the python project, it will get stuck in the while 1 loop, thereby blocking the execution of other codes.

Simple loops but threaded

Thread is a concept in computer science. The small program with its own instructions is executed by the process and managed independently, which can solve the blocking situation of our first method. Let's see how.

import time
import threading
​
def task():
    print("Job Completed!")
​
def schedule():
    while 1:
        task()
        time.sleep(10)
​
# makes our logic non blocking
thread = threading.Thread(target=schedule)
thread.start()

After the thread is started, its underlying logic cannot be modified by the main thread, so we may need to add resources, through which the program can check specific scenarios and execute logic based on them.

Schedule Library

Earlier, I said that using a while loop for scheduling looks ugly, and the scheduling library can solve this problem.

import schedule
import time
​
def task():
    print("Job Executing!")
​
# for every n minutes
schedule.every(10).minutes.do(task)
​
# every hour
schedule.every().hour.do(task)
​
# every daya at specific time
schedule.every().day.at("10:30").do(task)
​
# schedule by name of day
schedule.every().monday.do(task)
​
# name of day with time
schedule.every().wednesday.at("13:15").do(task)
​
while True:
    schedule.run_pending()
    time.sleep(1)

As you can see, this way we can create multiple scheduling plans effortlessly. I particularly like the way of creating jobs and Method Chaining. On the other hand, this fragment has a while loop, which means the code is blocked, but I believe you already know what can help us solve this problem.

Python Crontab

The crontab utility in Liunx is an easy-to-use and widely accepted scheduling solution. The Python library python-crontab provides an API to use the CLI tools in Python. In crontab, a timing schedule is described using the unix-cron string format ( *), which is a line of a set of five values, which indicates that when the job should be executed, python-crontab will be in the file The plan written in crontab is converted to write programming method.

from crontab import CronTab
​
cron = CronTab(user='root')
​
job = cron.new(command='my_script.sh')
​
job.hour.every(1)
cron.write()

python-crontab does not automatically save the plan, you need to execute the write() method to save the plan. There are more features, I strongly recommend you to check their documentation.

RQ Scheduler

Some tasks cannot be executed immediately, so we need to create task queues and pop tasks based on queue systems such as LIFO or FIFO. python-rq allows us to do this, using Redis as a proxy to queue jobs. The entry of the new job is stored as a hash map with information, such as created_at, enqueued_at, origin, data, description.

Queuing tasks are executed by a program called worker. The workers also have an entry in the Redis cache, which is responsible for dequeuing tasks and updating the task status in Redis. Tasks can be queued when needed, but to schedule them, we need rq-scheduler.

from rq_scheduler import Scheduler
​
queue = Queue('circle', connection=Redis())
scheduler = Scheduler(queue=queue)
​
scheduler.schedule(
    scheduled_time=datetime.utcnow(), # Time for first execution, in UTC timezone
    func=func,                     # Function to be queued
    args=[arg1, arg2],             # Arguments passed into function when executed
    kwargs={'foo': 'bar'},         # Keyword arguments passed into function when executed
    interval=60,                   # Time before the function is called again, in seconds
    repeat=None,                     # Repeat this number of times (None means repeat forever)
    meta={'foo': 'bar'}            # Arbitrary pickleable data on the job itself
)

RQ worker (RQ worker) must be started separately in the terminal or via python-rq worker. Once the task is triggered, it can be seen in the work terminal, and a separate function callback can be used in both success and failure scenarios.

Conclusion

There are also some libraries for scheduling, but here, I have discussed the most common libraries. It is worth mentioning that Celery, another advantage of celery is that users can choose between multiple agents. I am grateful for you to read till the end. You can also check out my other articles. cheers!

Translation source: https://python.plainenglish.io/5-ways-to-schedule-jobs-in-python-99de8a80f28e

Click to follow and learn about Huawei Cloud's fresh technology for the first time~