django + celery + redis to implement scheduled tasks

foreword

Some time ago, there was a requirement at work to change the original timed task to use celery. The scheduled task in the original project is the apscheduler used. However, there are few online tutorials for django and celery, and the version does not correspond. Finally, according to the online tutorial + carefully read the official documentation of celery many times, it finally got through.

Now do a tutorial/record to quickly implement django+celery timing tasks.
The basic knowledge and installation of these three components will not be repeated. Go directly to the configuration.

Version

Django==3.1.4
celery==5.0.5
redis==3.5.3

Thought steps

The ideas and steps for configuring celery timing tasks are mainly as follows

1. Create celery instance
2. Configuration tasks
3. Write a task function
4. Start waker and beat
5. store the result

Directory level (for reference)

 django_demo                    # 项目根目录
    ├── scheduler              # 这是一个app
    │   ├── __init__.py
    │   ├── celery.py            # 实例化celery并指定config
    │   ├── config.py           # celery的配置文件
    │   └── tasks.py             # 任务函数

1. celery.py

 from __future__ import absolute_import
from celery import Celery

app = Celery("scheduler", broker="redis://:12345@localhost:6379/1", backend="redis://:12345@localhost:6379/2", include=["scheduler.tasks"])

app.config_from_object("scheduler.config")

This file is used to instantiate celery and specify broker and backend as redis (writable configuration file)
include refers to the task file
app.config_from_object() specifies the configuration of celery

2. config.py

 from __future__ import absolute_import

from datetime import timedelta
CELERY_TIMEZONE = "Asia/Shanghai"
CELERY_ENABLE_UTC = True

CELERYBEAT_SCHEDULE = {
    "test": {                                 #任务名，用于开发人员识别
        "task": "scheduler.tasks.test",       #task指向任务函数
        "schedule": timedelta(seconds=2),     #调度时间 还可以使用crontab
        "args": ()                            #没有参数可以不写
    },
}

This file can write some celery configuration, and load timed tasks through CELERYBEAT_SCHEDULE.
Timed time can be used crontab.
For example: crontab(hour="*/24") means that it can be executed every 24 hours or you can specify a time. For details, see how to use crontab

3. tasks.py

 from scheduler.celery import app

@app.task()
def test():
    print("hello")
    return None

import app must be imported from the instantiated celery.py. If the instantiation file is not called celery.py, or the object is not called app, it needs to be changed accordingly.
Task functions need to be decorated with @app.task(). If no return value is required, return can be omitted.

4. Terminal command to start worker and beat

You need to open 2 terminals here, or you can write them into a shell script together in liunx

 celery -A scheduler.celery beat -l info  # 启动beat
celery -A scheduler.celery worker -l info

scheduler.celery is set according to the directory and file, please change it according to your own project
-l info : output log when running

If it is started under windows, the worker command needs to be replaced with

-P eventlet : it is to start the worker under windows, and delete it from other systems
(If you are prompted that eventlet is not available, use pip to install eventlet)

 celery -A scheduler.celery worker -l info -P eventlet  # windows下启动worker

The above is a simple configuration of celery timing tasks

---

---

---

In a work project, the code needs to be as concise as possible, the configuration needs to be placed in the same general configuration file, and the project is divided into development and online environments, so my django configuration in the end is like this:

 django_demo                        # 项目根目录
    ├── settings
    |    ├── base.py                #基础设置
    |    ├── develop.py          #开发环境设置          
    ├── scheduler              # 这是一个app
    │   ├── __init__.py
    │   ├── celery.py            # 实例化celery并指定config
    │   └── tasks.py             # 任务函数

I wrote the configuration in config.py above into the settings file in django.

1. settings:

• develop.py:

 BROKER_URL = redis_url
CELERY_RESULT_BACKEND = redis_url

In the development environment, configure the broker and backend used for development. Note that the variable name cannot be wrongly written, otherwise Celery will not recognize it.
redis_url is our redis port, I wrote it elsewhere to facilitate the management of all components.
It can also be written like this:

 BROKER_URL = "redis://:12345@localhost:6379/1"
CELERY_RESULT_BACKEND = "redis://:12345@localhost:6379/1"

• base.py:
  Here I wrote the scheduling of the task into the base configuration of the project.

   # celery配置及任务配置
  CELERY_TIMEZONE = "Asia/Shanghai"
  CELERY_ENABLE_UTC = True
  
  CELERYBEAT_SCHEDULE = {
    "job1": {
        "task": "scheduler.tasks.job1",
        "schedule": crontab(minute=0, hour=3),
    },
    "job2": {
        "task": "scheduler.tasks.job2",
        "schedule": crontab(hour="*/24"),
    },
  }

2. celery.py

 from __future__ import absolute_import

import os

from celery import Celery

profile = os.environ.setdefault("PROFILE", "production")
os.environ.setdefault("DJANGO_SETTINGS_MODULE", f"demo.settings.{profile}")

app = Celery("scheduler", include=["scheduler.jobs"])
app.config_from_object("django.conf:settings")

1. In order to facilitate the difference between development and online, the overall environment is first set up.
2. profile and os.environ.setdefault are settings for the project environment.
3. The main body of celery is still app and app.config_from_object.

It can be found that I have fewer variables in Celery() here and are more concise. This is because I have written these elements into the settings of django. When celery is instantiated, it will be based on the "django" I configured. .conf:settings" to find the corresponding value.

3.jobs.py

 from scheduler.celery import app

@app.task()
def job1():
    pass

def job2():
    pass

Summarize:

Celery is relatively easy to implement timed tasks. It is mainly composed of three parts: instantiation of celery, configuration and task configuration, and task function. About the configuration can be adaptable, write anywhere. The key is that when the three files are referenced, they do not need to be confused.
One is when celery is instantiated, app.config_from_object() pay attention to write our configuration path, one is the task function path when configuring timed tasks in the configuration file, and the other is the decorator of the task function @app.task() must be an instance Objects in Celery.

Show the output of the scheduled task being executed.
local is the worker started
local2 is the start beat

In the beat terminal, when running, it will always print that the sending task is successful.
The worker will always output that the task is successfully received, and return the return value and the output in the task function.

refer to

• Celery official documentation
• Implementation of django+celery timed task nanny level tutorial under win10