Dolphinscheduler 输出变量解说

头像
journey
     北京
    阅读 9 分钟

    1、Shell 脚本中，单引号 (')、双引号 (") 和反引号 (`)使用

    在 Shell 脚本中，单引号 (')、双引号 (") 和反引号 (`) 各自有不同的作用和用法。理解它们的区别和用法对于编写和调试 Shell 脚本非常重要

    1.1、单引号 (')

    • 用途：完全引用，用于保护字符串中的所有字符，不进行变量替换或命令替换
    • 特性：引号内的任何字符都将原样输出，不会进行任何解释
    VAR="world"
echo 'Hello, $VAR'  # 输出：Hello, $VAR

    1.2、双引号 (")

    • 用途：部分引用，用于保护字符串中的大部分字符，但允许变量替换和命令替换
    • 特性：引号内的变量和命令会被解释，其余字符原样输出
    VAR="world"
echo "Hello, $VAR"  # 输出：Hello, world

    1.3、反引号 (`)

    • 用途：命令替换，用于执行引号内的命令，并将命令的输出作为结果返回
    • 特性：引号内的命令会被执行，结果会替换反引号中的内容
    • 注意：反引号是旧的命令替换方式，更推荐使用 $()
    DATE=`date`
echo "Current date and time: $DATE"  # 输出当前日期和时间

    1.4、推荐使用 $() 进行命令替换

    • 更现代和可读性更好
    • 可以嵌套使用，而反引号嵌套使用比较困难
    DATE=$(date)
echo "Current date and time: $DATE"  # 输出当前日期和时间

# 嵌套命令替换示例
OUTER=$(echo "Outer $(echo "Inner")")
echo $OUTER  # 输出：Outer Inner

    1.5、使用场景

    • 单引号：当你不希望字符串中的任何内容被解释时使用，例如正则表达式、特殊字符等
    • 双引号：当你希望字符串中包含变量或命令替换时使用
    • 反引号/$()：当你需要执行命令并使用其输出时使用

    2、Dolphinscheduler中的输出变量和文件传递

    2.1、Shell输出变量

    2.1.1、流程如下

    image.png

    2.1.2、任务配置

    taskA
    image.png

    echo 'taskA'
echo "#{setValue(linesNum=${lines_num})}"
echo '${setValue(words=20)}'

    注意 : 这里${lines_num}其实直接就是通过Worker进行变量进行替换的

    taskB
    image.png

    echo 'taskB'
echo ${linesNum}
echo ${words}

    2.1.3、结果输出

    主要看taskB的输出

    }
[INFO] 2024-07-05 10:09:54.539 +0800 - Success initialized task plugin instance successfully
[INFO] 2024-07-05 10:09:54.539 +0800 - Set taskVarPool: [{"prop":"linesNum","direct":"IN","type":"VARCHAR","value":"100"},{"prop":"words","direct":"IN","type":"VARCHAR","value":"20"}] successfully
[INFO] 2024-07-05 10:09:54.539 +0800 - ***********************************************************************************************
[INFO] 2024-07-05 10:09:54.539 +0800 - *********************************  Execute task instance  *************************************
[INFO] 2024-07-05 10:09:54.539 +0800 - ***********************************************************************************************
[INFO] 2024-07-05 10:09:54.540 +0800 - Final Shell file is: 
[INFO] 2024-07-05 10:09:54.540 +0800 - ****************************** Script Content *****************************************************************
[INFO] 2024-07-05 10:09:54.540 +0800 - #!/bin/bash
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /etc/profile
export HADOOP_HOME=${HADOOP_HOME:-/home/hadoop-3.3.1}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
export SPARK_HOME=${SPARK_HOME:-/home/spark-3.2.1-bin-hadoop3.2}
export PYTHON_HOME=${PYTHON_HOME:-/opt/soft/python}
export HIVE_HOME=${HIVE_HOME:-/home/hive-3.1.2}
export FLINK_HOME=/home/flink-1.18.1
export DATAX_HOME=${DATAX_HOME:-/opt/soft/datax}
export SEATUNNEL_HOME=/opt/software/seatunnel
export CHUNJUN_HOME=${CHUNJUN_HOME:-/opt/soft/chunjun}

export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$SEATUNNEL_HOME/bin:$CHUNJUN_HOME/bin:$PATH
echo 'taskB'
echo 100
echo 20
[INFO] 2024-07-05 10:09:54.540 +0800 - ****************************** Script Content *****************************************************************
[INFO] 2024-07-05 10:09:54.540 +0800 - Executing shell command : sudo -u root -i /tmp/dolphinscheduler/exec/process/root/13850571680800/14171397340576_3/1961/1454/1961_1454.sh
[INFO] 2024-07-05 10:09:54.544 +0800 - process start, process id is: 588336
[INFO] 2024-07-05 10:09:56.544 +0800 -  -> 
    taskB
    100
    20
[INFO] 2024-07-05 10:09:56.546 +0800 - process has exited. execute path:/tmp/dolphinscheduler/exec/process/root/13850571680800/14171397340576_3/1961/1454, processId:588336 ,exitStatusCode:0 ,processWaitForStatus:true ,processExitValue:0

    2.2、Shell文件传递

    2.2.1、流程如下

    image.png

    2.2.2、任务配置

    fileUploadTask
    image.png

    echo 'fileUploadTask'

mkdir -p data/test1 data/test2
echo "test1 message" >> data/test1/text.txt
echo "test2 message" >> data/test2/text.txt

tree .

    fileDownloadTask
    image.png

    echo 'fileDownloadTask'

cat input_dir/test1/text.txt
cat input_dir/test2/text.txt

    2.2.3、结果输出

    fileDownloadTask

    [INFO] 2024-07-05 11:11:08.160 +0800 - Success initialized task plugin instance successfully
[INFO] 2024-07-05 11:11:08.160 +0800 - Set taskVarPool: [{"prop":"fileUploadTask.file-text","direct":"IN","type":"FILE","value":"DATA_TRANSFER/20240705/14171986797856/2_1962/fileUploadTask_1455_text.txt"},{"prop":"fileUploadTask.dir-data","direct":"IN","type":"FILE","value":"DATA_TRANSFER/20240705/14171986797856/2_1962/fileUploadTask_1455_data_ds_pack.zip"}] successfully
[INFO] 2024-07-05 11:11:08.160 +0800 - ***********************************************************************************************
[INFO] 2024-07-05 11:11:08.160 +0800 - *********************************  Execute task instance  *************************************
[INFO] 2024-07-05 11:11:08.160 +0800 - ***********************************************************************************************
[INFO] 2024-07-05 11:11:08.160 +0800 - Final Shell file is: 
[INFO] 2024-07-05 11:11:08.160 +0800 - ****************************** Script Content *****************************************************************
[INFO] 2024-07-05 11:11:08.160 +0800 - #!/bin/bash
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /etc/profile
export HADOOP_HOME=${HADOOP_HOME:-/home/hadoop-3.3.1}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
export SPARK_HOME=${SPARK_HOME:-/home/spark-3.2.1-bin-hadoop3.2}
export PYTHON_HOME=${PYTHON_HOME:-/opt/soft/python}
export HIVE_HOME=${HIVE_HOME:-/home/hive-3.1.2}
export FLINK_HOME=/home/flink-1.18.1
export DATAX_HOME=${DATAX_HOME:-/opt/soft/datax}
export SEATUNNEL_HOME=/opt/software/seatunnel
export CHUNJUN_HOME=${CHUNJUN_HOME:-/opt/soft/chunjun}

export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$SEATUNNEL_HOME/bin:$CHUNJUN_HOME/bin:$PATH
echo 'fileDownloadTask'

cat input_dir/test1/text.txt
cat input_dir/test2/text.txt
[INFO] 2024-07-05 11:11:08.161 +0800 - ****************************** Script Content *****************************************************************
[INFO] 2024-07-05 11:11:08.161 +0800 - Executing shell command : sudo -u root -i /tmp/dolphinscheduler/exec/process/root/13850571680800/14171986797856_2/1962/1456/1962_1456.sh
[INFO] 2024-07-05 11:11:08.164 +0800 - process start, process id is: 590323
[INFO] 2024-07-05 11:11:10.164 +0800 -  -> 
    fileDownloadTask
    test1 message
    test2 message
[INFO] 2024-07-05 11:11:10.166 +0800 - process has exited. execute path:/tmp/dolphinscheduler/exec/process/root/13850571680800/14171986797856_2/1962/1456, processId:590323 ,exitStatusCode:0 ,processWaitForStatus:true ,processExitValue:0

    2.3、SQL任务输出变量

    模式从SqlTask任务进行结果的输出，ShellTask对结果使用

    2.3.1、流程如下

    image.png

    2.3.2、任务配置

    sqlOutVarTask
    image.png

    select user_name as userNameList from t_ds_user

    readOutVarTask
    image.png

    echo 'readOutVarTask'
echo ${userNameList}

    2.3.3、结果输出

    readOutVarTask

    [INFO] 2024-07-05 11:19:00.294 +0800 - Success initialized task plugin instance successfully
[INFO] 2024-07-05 11:19:00.294 +0800 - Set taskVarPool: [{"prop":"userNameList","direct":"IN","type":"LIST","value":"[\"admin\",\"qiaozhanwei\",\"test\"]"}] successfully
[INFO] 2024-07-05 11:19:00.294 +0800 - ***********************************************************************************************
[INFO] 2024-07-05 11:19:00.294 +0800 - *********************************  Execute task instance  *************************************
[INFO] 2024-07-05 11:19:00.294 +0800 - ***********************************************************************************************
[INFO] 2024-07-05 11:19:00.295 +0800 - Final Shell file is: 
[INFO] 2024-07-05 11:19:00.295 +0800 - ****************************** Script Content *****************************************************************
[INFO] 2024-07-05 11:19:00.295 +0800 - #!/bin/bash
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /etc/profile
export HADOOP_HOME=${HADOOP_HOME:-/home/hadoop-3.3.1}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/soft/hadoop/etc/hadoop}
export SPARK_HOME=${SPARK_HOME:-/home/spark-3.2.1-bin-hadoop3.2}
export PYTHON_HOME=${PYTHON_HOME:-/opt/soft/python}
export HIVE_HOME=${HIVE_HOME:-/home/hive-3.1.2}
export FLINK_HOME=/home/flink-1.18.1
export DATAX_HOME=${DATAX_HOME:-/opt/soft/datax}
export SEATUNNEL_HOME=/opt/software/seatunnel
export CHUNJUN_HOME=${CHUNJUN_HOME:-/opt/soft/chunjun}

export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$SEATUNNEL_HOME/bin:$CHUNJUN_HOME/bin:$PATH
echo 'readOutVarTask'
echo ["admin","qiaozhanwei","test"]
[INFO] 2024-07-05 11:19:00.295 +0800 - ****************************** Script Content *****************************************************************
[INFO] 2024-07-05 11:19:00.295 +0800 - Executing shell command : sudo -u root -i /tmp/dolphinscheduler/exec/process/root/13850571680800/14172048617888_1/1963/1458/1963_1458.sh
[INFO] 2024-07-05 11:19:00.299 +0800 - process start, process id is: 590781
[INFO] 2024-07-05 11:19:02.299 +0800 -  -> 
    readOutVarTask
    [admin,qiaozhanwei,test]
[INFO] 2024-07-05 11:19:02.301 +0800 - process has exited. execute path:/tmp/dolphinscheduler/exec/process/root/13850571680800/14172048617888_1/1963/1458, processId:590781 ,exitStatusCode:0 ,processWaitForStatus:true ,processExitValue:0

    如感兴趣，点赞加关注，谢谢!!!

    大数据调度器java
    journey
    32 声望21 粉丝
    « 上一篇
    HDFS NameNode HA固定一个NN为Active
    下一篇 »
    Java 8 总结

    引用和评论

    推荐阅读
    头像
    Dolphinscheduler之quartz

    journey阅读 87

    头像
    在Java程序中监听mysql的binlog

    huan1993赞 10阅读 1.9k

    头像
    Jerry和您聊聊Chrome开发者工具

    注销赞 9阅读 4.5k评论 3

    头像
    Bitmap 和 布隆过滤器傻傻分不清？你这不应该啊

    程序员小富赞 8阅读 1.6k评论 1

    头像
    Just for fun——迅速写完快速排序

    ufdf赞 3阅读 2.6k

    头像
    Spring 实现 3 种异步流式接口，干掉接口超时烦恼

    程序员小富赞 5阅读 1.7k

    头像
    💢线上高延迟请求排查

    crossoverJie赞 6阅读 802评论 5

    0 条评论
    得票最新