Dolphinscheduler ShellTask CPU和Memory 使用及BUG修复

ShellTask默认情况下设置CPU配额和最大内存是不起作用的

1、系统支持

Ubuntu 或 Debian 系统：

sudo apt update
sudo apt install systemd

CentOS 或 RHEL 系统：
sudo yum install systemd

Fedora 系统 :
sudo dnf install systemd

1、开启

common.properties

# Task resource limit state，这里默认是false
task.resource.limit.state=true

2、原理

如果是flase的话，其实下面的逻辑是不会走的，所以要设置为true才能玩耍
org.apache.dolphinscheduler.plugin.task.api.shell.BaseLinuxShellInterceptorBuilder#bootstrapCommandInSudoMode

private List<String> bootstrapCommandInSudoMode() {
    // TODO 如果task.resource.limit.state为false，这里的逻辑不会走，也不会走CPU和内存的限制
    if (PropertyUtils.getBoolean(AbstractCommandExecutorConstants.TASK_RESOURCE_LIMIT_STATE, false)) {
        return bootstrapCommandInResourceLimitMode();
    }
    List<String> bootstrapCommand = new ArrayList<>();
    bootstrapCommand.add("sudo");
    if (StringUtils.isNotBlank(runUser)) {
        bootstrapCommand.add("-u");
        bootstrapCommand.add(runUser);
    }
    bootstrapCommand.add("-i");
    bootstrapCommand.add(shellAbsolutePath().toString());
    return bootstrapCommand;
}

private List<String> bootstrapCommandInResourceLimitMode() {
        List<String> bootstrapCommand = new ArrayList<>();
        bootstrapCommand.add("sudo");
        bootstrapCommand.add("systemd-run");
        bootstrapCommand.add("-q");
        bootstrapCommand.add("--scope");

        if (cpuQuota == -1) {
            bootstrapCommand.add("-p");
            bootstrapCommand.add("CPUQuota=");
        } else {
            bootstrapCommand.add("-p");
            bootstrapCommand.add(String.format("CPUQuota=%s%%", cpuQuota));
        }

        // use `man systemd.resource-control` to find available parameter
        if (memoryQuota == -1) {
            bootstrapCommand.add("-p");
            bootstrapCommand.add(String.format("MemoryLimit=%s", "infinity"));
        } else {
            bootstrapCommand.add("-p");
            bootstrapCommand.add(String.format("MemoryLimit=%sM", memoryQuota));
        }

        bootstrapCommand.add(String.format("--uid=%s", runUser));
        bootstrapCommand.add(shellAbsolutePath().toString());
        return bootstrapCommand;
    }

org.apache.dolphinscheduler.plugin.task.api.shell.BaseLinuxShellInterceptorBuilder#bootstrapCommandInResourceLimitMode

private List<String> bootstrapCommandInResourceLimitMode() {
    List<String> bootstrapCommand = new ArrayList<>();
    bootstrapCommand.add("sudo");
    bootstrapCommand.add("systemd-run");
    bootstrapCommand.add("-q");
    bootstrapCommand.add("--scope");

    if (cpuQuota == -1) {
        bootstrapCommand.add("-p");
        bootstrapCommand.add("CPUQuota=");
    } else {
        bootstrapCommand.add("-p");
        bootstrapCommand.add(String.format("CPUQuota=%s%%", cpuQuota));
    }

    // use `man systemd.resource-control` to find available parameter
    if (memoryQuota == -1) {
        bootstrapCommand.add("-p");
        bootstrapCommand.add(String.format("MemoryLimit=%s", "infinity"));
    } else {
        bootstrapCommand.add("-p");
        bootstrapCommand.add(String.format("MemoryLimit=%sM", memoryQuota));
    }

    bootstrapCommand.add(String.format("--uid=%s", runUser));
    // TODO 这个是我自己加的，官网dev看还没有修复，赶紧提PR吧
    bootstrapCommand.add(shellAbsolutePath().toString());
    return bootstrapCommand;
}

3、执行

新建任务

top查看运行脚本

top -b -n 1 -w 512

PID           USER      PR        NI     VIRT    RES     SHR S  %CPU  %MEM    TIME+ COMMAND
3977721       root      20        0      12992   3472   2964 S   0.0   0.0   0:00.00 1870_1287.sh

查看日志的运行结果

[INFO] 2024-06-19 15:42:18.643 +0800 - ****************************** Script Content *****************************************************************
[INFO] 2024-06-19 15:42:18.643 +0800 - Executing shell command : sudo systemd-run -q --scope -p CPUQuota=15% -p MemoryLimit=10M --uid=root /tmp/dolphinscheduler/exec/process/root/13850571680800/13996695885600_3/1870/1287/1870_1287.sh
[INFO] 2024-06-19 15:42:18.647 +0800 - process start, process id is: 3977715
[INFO] 2024-06-19 15:42:22.648 +0800 -  -> 
    resourceLimitTask
[INFO] 2024-06-19 15:52:22.751 +0800 - process has exited. execute path:/tmp/dolphinscheduler/exec/process/root/13850571680800/13996695885600_3/1870/1287, processId:3977715 ,exitStatusCode:0 ,processWaitForStatus:true ,processExitValue:0

注意 : 如果上面的BUG不解，就会有如下的报错，提示 Command line to execute required，任务失败

[INFO] 2024-06-19 15:22:58.968 +0800 - ****************************** Script Content *****************************************************************
[INFO] 2024-06-19 15:22:58.968 +0800 - Executing shell command : sudo systemd-run -q --scope -p CPUQuota=15% -p MemoryLimit=10M --uid=root
[INFO] 2024-06-19 15:22:58.972 +0800 - process start, process id is: 3973824
[INFO] 2024-06-19 15:22:59.973 +0800 -  -> 
    Command line to execute required.
[INFO] 2024-06-19 15:22:59.974 +0800 - process has exited. execute path:/tmp/dolphinscheduler/exec/process/root/13850571680800/13996695885600_2/1868/1285, processId:3973824 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1

如感兴趣，点赞加关注，谢谢!!!