Application Practice of Workflow Engine in vivo Marketing Automation | Engine 03

Author: vivo Internet Server Team - Cheng Wangrong

This article is the 4th article of "Decryption of vivo Marketing Automation Technology". It analyzes the background of the introduction of workflow technology in marketing automation business and the introduction of workflow engine, and introduces the characteristics of several popular open source workflow engines in the industry. Design ideas and summary thinking in the process of project self-research and development.

"Vivo Marketing Automation Technology Decryption" series of articles:

Decryption of vivo marketing automation technology｜Introduction

How Design Patterns Improve the Scalability of vivo Marketing Automation Business | Engine 01

In-depth practice of state machine engine in vivo marketing automation | Engine 02

1. Business Background

The marketing automation platform can support the configuration of activity journey strategies for different user life cycles, and conduct differentiated marketing reach solutions according to different activity behaviors triggered by users. At the same time, there are different business processing processes (such as approval process and business flow) in the specific execution process of various types of activities.

The business process is complex and diverse, and the requirements change frequently. There are the following pain points in the project development process:

1. Long project delivery cycle: A complete business process needs to be iterated by version from scratch, resulting in long development time and high cost.
2. Repetitive development and testing of functions: There are many common processes between businesses, resulting in a lot of repetitive development and testing work, which is inefficient.
3. High maintenance cost: With the gradual development of the project business, the business process is gradually accumulated, the maintainability is reduced, and the system change affects the whole body.

How to separate the business logic from the control flow and let the production and research personnel focus more on the realization of the business is a key problem that needs to be solved. In the traditional OA field, the proven business process management solution - Workflow (Workflow) is used. Workflow is a set of industrial-grade solutions, developed by the Workflow Management Consortium (WfMC) to a series of standards.

2. Introduction to Workflow

2.1 Workflow Definition

Workflow - the abstraction of the business rules between the workflow and its operation steps, modeling the work organization logic and rules in the process, and handing it over to the computer for automatic processing.

The essential idea of workflow is: the process of instantiating real activities through predefined workflow templates . Simply put, it is a template for configuring a process in a preset format or visually (such as a running process template for sharing activities). When using it, a process instance object is constructed from the template, and the activity running tracking and backtracking are completed through the instance object.

2.2 Workflow Reference Model

The WfMC Workflow Management Alliance has formulated a reference model for workflow, the core of which is the intermediate workflow engine. The workflow engine provides process definition tools (interface 1), provides users with information query (interface 2), and invokes external applications (interface 1). 3), the ability to integrate other workflows (interface 4) and monitoring management (interface 5). For most workflow products, the focus is on the implementation of Interface 1 and Interface 2.

2.3 Key Features of Workflow Engine

• Process visualization

Provides visual process construction, process view viewing capabilities, and real-time observation task running capabilities.

• Business orchestration and reuse

Componentizing public services can support the free arrangement of tasks and freely build different processes for suitable services.

• Business and Control Separation

The task of process control (such as flow, judgment, loop, retry, etc.) is handed over to the workflow, so that users can focus on the core business logic.

2.4 Types of Workflow Engines

There is no specific standard for the type of workflow. According to the characteristics of the process task node, it can be divided into:

• sequential workflow

Sequential workflow works like a specific flow chart. After the previous process task is completed, the next process task is entered in turn, and the process is irreversible.

• State Machine Workflow

The state machine workflow focuses on the state of the process task. The factors that drive the task state to change are generally external events, that is, the event-driven way, which drives the task node to run from one state to another state, and the nodes are reversible.

• rule-driven workflow

Focusing on the operating rules of nodes, and the execution of workflows based on business rules, rule-driven workflows are very useful when dealing with various projects with clear goals but different levels of "rules" or specifications.

It can be seen that different types of workflows are not completely separated, and the state machine workflow can also combine conditions and rules to perform the process of operating node transitions. In software development, a combination of state machines and rule-driven workflows is generally considered.

2.5 Differences between Workflow Engines and State Machines

In the previous article, we made a simple comparison between state machine and workflow engine . In fact, the two are not completely equivalent concepts:

• A state machine is a mathematical calculation model of system states and behaviors such as transitions and actions between these states, while a workflow is an abstract modeling of the overall workflow and its business logic and rules between operation steps.
• The state machine mode is event-driven, most of which trigger the automatic flow of states through external events; the workflow engine is more focused on describing the automatic flow after the completion of predefined process tasks, and the predictability will be stronger.
• From the perspective of the complexity of the applicable scenarios, the direct use of the state machine can clearly describe all possible states and events that lead to transitions, which is suitable for solving single-dimensional, low-complexity business problems, and is flexible and lightweight; The workflow engine is more suitable for complex business process management, solving process automation problems with higher complexity such as large-scale CRM, and focusing on improving the efficiency of the overall business process.
• The workflow engine can be constructed on the basis of the structural model of the state machine. In fact, many open source workflow engines are also implemented based on the state machine.

After understanding the basic features and usage scenarios of workflow, let's take a look at the more popular open source workflow engines.

3. Open source workflow engine

4. Self-developed design of workflow engine

4.1 Problems with Open Source Workflow Engines

• The biggest advantage of open source workflow is that it can be used out of the box with the help of open source resources and has comprehensive functions. Taking Activiti as an example, using Activiti 7.0 requires at least 20 tables to be introduced. Although it seems to be a non-invasive method, there is a certain cost in the process of system evolution and maintenance. Especially when there are many business process instances, developers need to have deeper control over the table logic.
• Due to the objective uniqueness of the business, as a business process component, it generally needs to be re-developed and adapted according to its own business. For example, it is necessary to control the user role permissions of process nodes according to its own organizational structure; plug-in its own business capabilities, add it to the workflow configuration, and intercept callbacks.

4.2 Core design ideas of self-developed engine

4.2.1 Engine core module

Returning to the essence of workflow, workflow is the process of instantiating real activities through predefined process templates. A basic workflow engine mainly includes three core parts:

• Process template creation

According to business rules and logic, create process templates and set the operation and change paths of each node. Based on template creation, process designers, plug-in nodes, diverse template file formats, template persistence, etc. can be extended.

• Process instance release

According to the process template, a process instance is created, and the relationship between the process template and the process instance is similar to the relationship between the class and the object. For example, the work order system administrator defines an approval flow template (process template), and the user clicks to create a work order (process instance). Based on the publishing of process instances, real-time observation of instances, backtracking of node transition records, persistence of instance status, failure retry, transaction control, etc. can be extended.

• task process execution

After the process instance is created, the process instance only needs to independently execute the tasks of each instance according to the definition of the process template. Different instances do not affect each other and complete the life cycle of each instance.

4.2.2 Engine core design

① When the application container starts, load the process engine environment configuration, including the parser structure, process engine context, process definition file path, etc.

② Read the predetermined process definition file, parse the process node, build the execution context, and put the process node in the memory cache.

③ The business side creates a process, starts a new process instance, and binds the business process and the process instance at the same time.

④ Run each node of the process instance, and save each process node persistently.

4.3 Specific practice

① Engine core services.

The main external interface of engine operation, including starting process instances, and obtaining related process definition templates, process instances, and process node services.

 public interface FlowEngine {
 
    /**
     * 根据流程定义key，参数列表启动流程实例
     *
     */
    FlowInstance startInstance(String processDefKey, Map<String, Object> args);
 
    /**
     * 根据流程定义主键ID，参数列表执行流程任务（推动流程自动流转）
     * 统一事务控制
     */
    void execInstance(Long instanceId, Map<String, Object> args) throws FlowAuthorityException;
 
     /**
     * 获取流程定义process服务
     *
     */
    ProcessService process();
 
    /**
     * 获取流程实例服务
     *
     */
    InstanceService instance();
 
    /**
     * 获取任务节点服务
     *
     */
    TaskService task();
}

② Process definition service.

It is mainly for the creation and release of process definition templates, and different creation methods can be supported according to specific implementation classes.

 public interface ProcessService {  
   /**
     * 创建流程定义模板
     *
     */
    void create(String definition);
 
   /**
     * 发布流程定义模板
     *
     */
    void deploy(String fileName);
 
    /**
     * 获取流程key对应的流程定义
     */
    FlowProcess getProcessByDefKey(String processDefKey);
}

③ Process instance service.

Provides an entry for process instance creation persistence and process instance execution.

 public interface InstanceService {
    /**
     * 创建流程实例
     *
     */
    FlowInstance createInstance(FlowProcess process, Map<String, Object> args);
 
    /**
     * 执行流程实例
     *
     * @param instanceId 流程实例id
     */
    void exec(Long instanceId);
 
    /**
     * 根据id获取流程实例
     *
     * @param instanceId
     * @return
     */
    FlowInstance getById(Long instanceId);
}

④ Process task node service.

Provides the creation and query of each task of the process node.

 public interface TaskService {
 
    /**
     * 根据任务模型、执行对象创建新的任务
     *
     */
    FlowTask createTask(TaskModel taskModel, Execution execution);
 
    /**
     * 完成任务
     *
     */
    FlowTask complete(Long taskId,  Map<String, Object> args);
 
    /**
     * 获取流程实例中正在进行的任务
     *
     */
    FlowTask getActiveTask(Long instanceId);
 
    /**
     * 获取流程实例上一个已完成的任务
     *
     */
    FlowHistTask getLastDoneTask(Long instanceId);
}

The core method is

FlowEngine#startInstanceByKey, start the process instance. Based on the process definition, create a process instance object.

FlowEngine#execInstance, executes the task of the process instance, and promotes the automatic flow of the process instance according to the node tasks defined by the process through the incoming context parameters (operator, operation variables, etc.).

4.4 Thinking and Expanding

• Process definition parsing performance.

Since the current design is to load and parse all process definition files when the application starts, if there are too many process definition files, it will affect the application startup speed. It can be optimized through multi-threaded parsing and lazy loading (parsing while in use).

• Process definition version compatibility.

Since the business process is not static, it will continue to iterate during the development of the project, and it is necessary to be compatible with the different previous processes.

• Process node plug-in and orchestration capabilities.

Extract common basic services to support plug-in and orchestration capabilities for drawing different processes.

• Process execution monitoring capabilities.

Embedding reports on the execution of process task nodes, and the system automatically monitors and alarms.

V. Summary

This paper analyzes the background of the introduction of the workflow engine, which drives the business logic to be separated from the control flow, allows the production and research team to focus more on the business, and solves the problem of low research and development efficiency.

The essential idea of workflow is the process of instantiating real activities through predefined workflow templates. Generally, it is required to have the basic capabilities of process visualization, business orchestration and reuse, and separation of business and control. Common workflows are divided into sequential workflows, state machine workflows, and rule-driven workflows. The most common open source workflow framework is state machine workflows, which use event-driven methods to drive processes.

At the same time, it briefly introduces the characteristics of several popular open source workflow engines in the industry, combined with the characteristics of open source workflow engines, and for diversified and frequently iterative business processes, starting from the essential idea of workflow, we develop our own research. Developed a set of lightweight workflow engine, shared the design ideas and summary thinking in the practice process.