Dolphinscheduler 中的参数及子流程变量池传递失效BUG

1、Dolphinscheduler中的参数

1. command_param 中的启动参数
2. 本地参数
3. 变量池
4. 全局参数
5. 项目参数
6. 内置参数(业务时间参数、系统内置参数、衍生内置参数)，衍生内置参数是在Worker端进行处理的

1.1、Master参数设置

org.apache.dolphinscheduler.server.master.runner.TaskExecutionContextFactory#createTaskExecutionContext

public TaskExecutionContext createTaskExecutionContext(TaskInstance taskInstance) throws TaskExecutionContextCreateException {
    ProcessInstance workflowInstance = taskInstance.getProcessInstance();

    ResourceParametersHelper resources =
            Optional.ofNullable(taskPluginManager.getTaskChannel(taskInstance.getTaskType()))
                    .map(taskChannel -> taskChannel.getResources(taskInstance.getTaskParams()))
                    .orElse(null);
    setTaskResourceInfo(resources);

    Map<String, Property> businessParamsMap = new HashMap<>();
    if (workflowInstance != null) {
        businessParamsMap = curingParamsService.preBuildBusinessParams(workflowInstance);
    }

    AbstractParameters baseParam = taskPluginManager.getParameters(ParametersNode.builder()
            .taskType(taskInstance.getTaskType()).taskParams(taskInstance.getTaskParams()).build());

    Map<String, Property> propertyMap = new HashMap<>();
    if (workflowInstance != null) {
        // TODO 主要是这里进行的封装
        propertyMap = curingParamsService.paramParsingPreparation(taskInstance, baseParam, workflowInstance);
    }

    TaskExecutionContext taskExecutionContext = TaskExecutionContextBuilder.get()
            .buildWorkflowInstanceHost(masterConfig.getMasterAddress())
            .buildTaskInstanceRelatedInfo(taskInstance)
            .buildTaskDefinitionRelatedInfo(taskInstance.getTaskDefine())
            // TODO 这里面设置的租户code
            .buildProcessInstanceRelatedInfo(taskInstance.getProcessInstance())
            .buildProcessDefinitionRelatedInfo(taskInstance.getProcessDefine())
            .buildResourceParametersInfo(resources)
            .buildBusinessParamsMap(businessParamsMap)
            // TODO 这里是重点
            .buildParamInfo(propertyMap)
            .create();

    if (taskInstance.getProcessInstance() == null) {
        taskExecutionContext.setTenantCode(taskInstance.getTenantCode());
    }
    
    if (workflowInstance != null) {
        setDataQualityTaskExecutionContext(taskExecutionContext, taskInstance, workflowInstance.getTenantCode());
    }

    setK8sTaskRelatedInfo(taskExecutionContext, taskInstance);
    return taskExecutionContext;
}

org.apache.dolphinscheduler.service.expand.CuringParamsServiceImpl#paramParsingPreparation

public Map<String, Property> paramParsingPreparation(@NonNull TaskInstance taskInstance,
                                                         @NonNull AbstractParameters parameters,
                                                         @NonNull ProcessInstance processInstance) {
        Map<String, Property> prepareParamsMap = new HashMap<>();

        // assign value to definedParams here
        // TODO 全局参数
        Map<String, String> globalParamsMap = setGlobalParamsMap(processInstance);
        // TODO 要明白，不一定里面都是IN参数
        Map<String, Property> globalParams = ParameterUtils.getUserDefParamsMap(globalParamsMap);

        // combining local and global parameters
        // TODO 将IN的local参数获取到
        Map<String, Property> localParams = parameters.getInputLocalParametersMap();

        // stream pass params
        // TODO 变量池
        parameters.setVarPool(taskInstance.getVarPool());
        Map<String, Property> varParams = parameters.getVarPoolMap();

        // if it is a complement,
        // you need to pass in the task instance id to locate the time
        // of the process instance complement
        Map<String, String> cmdParam = JSONUtils.toMap(processInstance.getCommandParam());
        String timeZone = cmdParam.get(Constants.SCHEDULE_TIMEZONE);

        // built-in params
        // TODO 内置参数 : 包含了业务时间参数 + 系统内置参数
        Map<String, String> builtInParams = setBuiltInParamsMap(taskInstance, timeZone);

        // project-level params
        // TODO 项目参数
        Map<String, Property> projectParams = getProjectParameterMap(taskInstance.getProjectCode());

        // TODO 内置参数
        if (MapUtils.isNotEmpty(builtInParams)) {
            prepareParamsMap.putAll(ParameterUtils.getUserDefParamsMap(builtInParams));
        }

        // TODO 项目参数
        if (MapUtils.isNotEmpty(projectParams)) {
            prepareParamsMap.putAll(projectParams);
        }

        // TODO 全局参数
        if (MapUtils.isNotEmpty(globalParams)) {
            prepareParamsMap.putAll(globalParams);
        }

        // TODO 变量池
        if (MapUtils.isNotEmpty(varParams)) {
            prepareParamsMap.putAll(varParams);
        }

        // TODO 本地参数
        if (MapUtils.isNotEmpty(localParams)) {
            prepareParamsMap.putAll(localParams);
        }

        // TODO 启动参数
        if (MapUtils.isNotEmpty(cmdParam)) {
            prepareParamsMap.putAll(parseWorkflowStartParam(cmdParam));
        }

        Iterator<Map.Entry<String, Property>> iter = prepareParamsMap.entrySet().iterator();
        while (iter.hasNext()) {
            Map.Entry<String, Property> en = iter.next();
            Property property = en.getValue();

            if (StringUtils.isNotEmpty(property.getValue())
                    && property.getValue().contains(Constants.FUNCTION_START_WITH)) {
                /**
                 *  local parameter refers to global parameter with the same name
                 *  note: the global parameters of the process instance here are solidified parameters,
                 *  and there are no variables in them.
                 */
                String val = property.getValue();
                // whether external scaling calculation is required
                // TODO 这个逻辑不会走
                if (timeFunctionNeedExpand(val)) {
                    val = timeFunctionExtension(taskInstance.getProcessInstanceId(), timeZone, val);
                } else {
                    // handle some chain parameter assign, such as `{"var1": "${var2}", "var2": 1}` should be convert to
                    // `{"var1": 1, "var2": 1}`
                    // TODO 其实就是变量的替换，${xx} 换成真正的内容
                    val = convertParameterPlaceholders(val, prepareParamsMap);
                }
                property.setValue(val);
            }
        }

        // put schedule time param to params map
        // TODO 如果调度时间存在 system.datetime也会放入到prepareParamsMap中
        Map<String, Property> paramsMap = preBuildBusinessParams(processInstance);
        if (MapUtils.isNotEmpty(paramsMap)) {
            prepareParamsMap.putAll(paramsMap);
        }
        return prepareParamsMap;
    }

1.2、Worker中参数的使用

org.apache.dolphinscheduler.plugin.task.api.utils.ParameterUtils#convertParameterPlaceholders

以ShellTask为示例 :

org.apache.dolphinscheduler.plugin.task.shell.ShellTask#handle

public void handle(TaskCallBack taskCallBack) throws TaskException {
        try {

            IShellInterceptorBuilder<?, ?> shellActuatorBuilder = ShellInterceptorBuilderFactory.newBuilder()
                    .properties(ParameterUtils.convert(taskExecutionContext.getPrepareParamsMap())) // TODO 这里就是要进行变量的替换
                    .appendScript(shellParameters.getRawScript());

            // TODO shell执行
            TaskResponse commandExecuteResult = shellCommandExecutor.run(shellActuatorBuilder, taskCallBack);
            // TODO 执行结果，退出状态码
            setExitStatusCode(commandExecuteResult.getExitStatusCode());
            // TODO 设置进程ID
            setProcessId(commandExecuteResult.getProcessId());
            // TODO shellCommandExecutor.getTaskOutputParams()这返回的是 output -> 123
            shellParameters.dealOutParam(shellCommandExecutor.getTaskOutputParams());
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            log.error("The current Shell task has been interrupted", e);
            setExitStatusCode(EXIT_CODE_FAILURE);
            throw new TaskException("The current Shell task has been interrupted", e);
        } catch (Exception e) {
            log.error("shell task error", e);
            setExitStatusCode(EXIT_CODE_FAILURE);
            throw new TaskException("Execute shell task error", e);
        }
    }

org.apache.dolphinscheduler.plugin.task.api.shell.BaseLinuxShellInterceptorBuilder#shellBody

private String shellBody() {
        if (CollectionUtils.isEmpty(scripts)) {
            return StringUtils.EMPTY;
        }
        String scriptBody = scripts
                .stream()
                .collect(Collectors.joining(System.lineSeparator()));
        scriptBody = scriptBody.replaceAll("\\r\\n", System.lineSeparator());

        // TODO 最终是通过变量进行替换的，这才是核心中的重点
        return ParameterUtils.convertParameterPlaceholders(scriptBody, propertyMap);
    }

其实核心的代码就是 :

ParameterUtils.convertParameterPlaceholders(scriptBody, propertyMap);
propertyMap 是谁呢 ? 就是 taskExecutionContext.getPrepareParamsMap()
taskExecutionContext.getPrepareParamsMap() 哪里来的内，其实就是Master在构建taskExecutionContext的时候将各类参数塞进去形成的一个map(org.apache.dolphinscheduler.service.expand.CuringParamsServiceImpl#paramParsingPreparation)

2、子流程变量池传递失效BUG

构造参数中增加一个 SubWorkflowLogicTask :

private SubWorkflowLogicTask subWorkflowLogicTask;

    public SubWorkflowAsyncTaskExecuteFunction(TaskExecutionContext taskExecutionContext,
                                               ProcessInstanceDao processInstanceDao,
                                               SubWorkflowLogicTask subWorkflowLogicTask) {
        this.taskExecutionContext = taskExecutionContext;
        this.processInstanceDao = processInstanceDao;
        this.subWorkflowLogicTask = subWorkflowLogicTask;
    }

org.apache.dolphinscheduler.server.master.runner.task.subworkflow.SubWorkflowAsyncTaskExecuteFunction#getAsyncTaskExecutionStatus

public @NonNull AsyncTaskExecutionStatus getAsyncTaskExecutionStatus() {
        // query the status of sub workflow instance
        if (subWorkflowInstance == null) {
            subWorkflowInstance = processInstanceDao.querySubProcessInstanceByParentId(
                    taskExecutionContext.getProcessInstanceId(), taskExecutionContext.getTaskInstanceId());
        }
        if (subWorkflowInstance == null) {
            log.info("The sub workflow instance doesn't created");
            return AsyncTaskExecutionStatus.RUNNING;
        }
        subWorkflowInstance = processInstanceDao.queryById(subWorkflowInstance.getId());

        // TODO 其实就是上报任务状态的时候，把变量池上报上去，所以需要加次句话
        subWorkflowLogicTask.getTaskParameters().setVarPool(subWorkflowInstance.getVarPool());

        if (subWorkflowInstance != null && subWorkflowInstance.getState().isFinished()) {
            return subWorkflowInstance.getState().isSuccess() ? AsyncTaskExecutionStatus.SUCCESS
                    : AsyncTaskExecutionStatus.FAILED;
        }
        return AsyncTaskExecutionStatus.RUNNING;
    }

Master逻辑任务请参考 : https://segmentfault.com/a/1190000044980743 这篇文章

如感兴趣，点赞加关注，谢谢!!!