Dolphinscheduler集成chunjun

1、配置

export CHUNJUN_HOME=/opt/dolphinscheduler/chunjun

因为 Dolphinscheduler 使用的是 start-chunjun脚本，start-chunjun需要如下修改 :

set -e

export CHUNJUN_HOME="$(cd "`dirname "$0"`"/..; pwd)"

# Find the java binary
if [ -n "${JAVA_HOME}" ]; then
  JAVA_RUN="${JAVA_HOME}/bin/java"
else
  if [ `command -v java` ]; then
    JAVA_RUN="java"
  else
    echo "JAVA_HOME is not set" >&2
    exit 1
  fi
fi

# 在这里修改，指定chunjun-clients.jar和其他lib下的jar包
JAR_DIR=$CHUNJUN_HOME/lib/chunjun-clients.jar:$CHUNJUN_HOME/lib/*
CLASS_NAME=com.dtstack.chunjun.client.Launcher

echo "ChunJun starting ..."
# 去掉nohup
$JAVA_RUN -cp $JAR_DIR $CLASS_NAME $@

2、将CHUNJUN放入到数据集成下面

/dolphinscheduler/dolphinscheduler-api/src/main/resources/task-type-config.yaml

task:
....
  dataIntegration:
    - 'DATAX'
    - 'CHUNJUN'
....

3、yarn-session

启动yarn-session，参考官网，注意一定要使用flink-1.12.7，因为纯均内置的是flink-1.12.7，其它版本会报错

cd $FLINK_HOME/bin
./yarn-session -t $CHUNJUN_HOME -d

如下配置 :

4、yarn-perjob(mysql2mysql为例)

JSON如下 ：

{
  "job": {
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "column": [
              {
                "name": "id",
                "type": "int"
              },
              {
                "name": "name",
                "type": "string"
              },
              {
                "name": "age",
                "type": "string"
              }
            ],
            "username": "root",
            "password": "root@123",
            "connection": [
              {
                "jdbcUrl": [
                  "jdbc:mysql://xx.xx.xx.xx:3306/journey?useSSL=false&allowPublicKeyRetrieval=true"
                ],
                "table": [
                  "chunjun_source"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "mysqlwriter",
          "parameter": {
            "username": "root",
            "password": "root@123",
            "connection": [
              {
                "jdbcUrl": "jdbc:mysql://xx.xx.xx.xx:3306/dest?useSSL=false&allowPublicKeyRetrieval=true",
                "table": [
                  "chunjun_sink"
                ]
              }
            ],
            "writeMode": "insert",
            "column": [
              {
                "name": "id",
                "type": "int"
              },
              {
                "name": "name",
                "type": "string"
              },
              {
                "name": "age",
                "type": "string"
              }
            ]
          }
        }
      }
    ],
    "setting": {
      "speed": {
        "channel": 1,
        "bytes": 0
      }
    }
  }
}

如感兴趣，点赞加关注，谢谢!!!