Add a background task module to the open source project go-gin-api

Task Management Interface (WEB)

Support the task management in the WEB interface, for example: add task , edit task edit task 1613349532614a0, enable/disable task , manually execute tasks etc.

The attributes of the task include:

• mission name

• execution way

  • SHELL
  • HTTP
• Expression ( /5 *)
• Order
• Timeout (seconds)
• number of retries
• Retry interval (seconds)

• Whether to notify the end of execution

  • No notice
  • Failure notification
  • End notice
  • Result keyword match notification
• state
• Remark

When the execution mode is HTTP , it supports selecting the request mode GET or POST ;

When setting the execution end notification, you can select the notification method email or Webhook ;

When setting up email notifications, you can enter multiple email addresses and split them;

When setting the result keyword matching notification, it supports inputting multiple keywords and dividing them;

After the task is added, the task data will be persisted to MySQL .

Task scheduler

Two open source components are referenced:

• robfig/cron
• jakecoffman/cron

Finally, I chose to use jakecoffman/cron , the latter is based on the former, for example, AddFunc() adds name parameters, and also adds RemoveJob(name string) support deleting specific tasks.

// AddFunc adds a func to the Cron to be run on the given schedule.
func (c *Cron) AddFunc(spec string, cmd func(), name string) {
    c.AddJob(spec, FuncJob(cmd), name)
}

...

// RemoveJob removes a Job from the Cron based on name.
func (c *Cron) RemoveJob(name string) {
    if !c.running {
        i := c.entries.pos(name)

        if i == -1 {
            return
        }

        c.entries = c.entries[:i+copy(c.entries[i:], c.entries[i+1:])]
        return
    }

    c.remove <- name
}

It can be used under simple encapsulation. The following is the encapsulation method. The specific implementation and use of the method are obtained from go-gin-api .

type Server interface {
    i()

    // Start 启动 cron 服务
    Start()

    // Stop 停止 cron 服务
    Stop()

    // AddTask 增加定时任务
    AddTask(task *cron_task_repo.CronTask)

    // RemoveTask 删除定时任务
    RemoveTask(taskId int)

    // AddJob 增加定时任务执行的工作内容
    AddJob(task *cron_task_repo.CronTask) cron.FuncJob
}

When calling Start() start the service, MySQL will be loaded into the scheduler.

Through the above methods, when adding, editing, enabling/disabling, and manually executing tasks WEB interface, the tasks in the scheduler can be dynamically managed.

Task executor

The task executor refers to the machine where the task is actually executed.

My idea is to use Kafka When the scheduler finds a task that needs to be executed, it writes the task information to Kafka of Topic , and the task executor subscribes to the related Topic obtain task information and then execute the task.

If the task execution mode is HTTP , then the task executor can be a group of clusters, which specialize in processing and calling HTTP tasks. Here it can be a consumer group (06134953261963), or it can be Consumer Group

If the execution mode of SHELL , then the task executor must be on the host where the script is located, which can be a consumer of a specific task.

If there are too many tasks, you can consider setting more Topic according to the business scenario.

In order to facilitate the demonstration in the project, it is not written to Kafka , only the log is recorded.

func (s *server) AddJob(task *cron_task_repo.CronTask) cron.FuncJob {
    return func() {
        s.taskCount.Add()
        defer s.taskCount.Done()

        msg := fmt.Sprintf("开始执行任务：(%d)%s [%s]", task.Id, task.Name, task.Spec)
        s.logger.Info(msg)
    }
}

Log directory: /logs/go-gin-api-cron.log .

summary

This article is purely an introduction, there are problems, criticisms and corrections are welcome.

The go-gin-api project is easy to install and can be used out of the box. Try to create a background task.

Recommended reading

• go-gin-api Two frequently asked writing questions
• go-gin-api adds two language pack related functions