Apache Linkis custom variable practice sharing

1. General description of requirements background Users hope that when writing code, they can define some common variables and then replace them when executing. For example, users will run the same sql in batches every day, and they need to specify the partition time of the previous day. If it is written based on sql, it will be more complicated. The system provides a run_date variable that will be very convenient to use.

The goal supports variable substitution of task code Supports custom variables, supports users to define custom variables in scripts and task parameters submitted to Linkis, supports simple +, - and other calculations Preset system variables: run_date, run_month, run_today and other system variables

2. The overall design is implemented in Entrance during the execution of the Linkis task. It is mainly implemented by intercepting and replacing the interceptor before the task is submitted and executed by Entrance. The variables and defined variables used in the task code are obtained through regular expressions, and The code is replaced by the initial value of the custom variable passed in by the task and becomes the final executable code.

2.1 Technical Architecture The overall architecture of custom variables is as follows. After the task is submitted, the interceptor will be replaced by the variable. First, all variables and expressions used in the code will be parsed, and then replaced with the system and user-defined initial values of variables, and finally the parsed code will be submitted to EngineConn for execution. So the underlying engine is already replaced code.
picture

3. Features
   The variable types supported by Linkis are divided into custom variables and system built-in variables. Internal variables are predefined by Linkis and can be used directly. Then different variable types support different calculation formats: String supports +, integer decimal supports +-*/, date supports +-.

3.1 Built-in variables The built-in variables currently supported are as follows:
picture

details:
1. run_date is the core built-in date variable, which supports user-defined date. If not specified, the default is the day before the current system time.
2. Definition of other derived built-in date variables: other date built-in variables are calculated relative to run_date. Once run_date changes, other variable values will also change automatically. Other date variables do not support setting initial values and can only be modified by modifying run_date. .
3. Built-in variables support more abundant usage scenarios: ${run_date-1} is the day before run_data; ${run_month_begin-1} is the first day of the previous month of run_month_begin, where -1 means minus one month.

3.2 Custom variables What are custom variables? User variables that are defined first and then used. User-defined variables temporarily support the definition of strings, integers, and floating-point variables. Strings support the + method, and integers and floating-point numbers support the +-*/ method. User-defined variables do not conflict with the set variable syntax supported by SparkSQL and HQL, but the same name is not allowed. How to define and use custom variables? as follows:

Defined in the code, specified before the task code

sql type definition method:

--@set f=20.1

The python/shell types are defined as follows:

@set f=20.1

Note: Only one variable can be defined on one line

The use is directly used in the code through {varName expression}, such as ${f*2}

3.3 Variable scope Custom variables in linkis also have scope, and the priority is that the variable defined in the script is greater than the Variable defined in the task parameters, and greater than the built-in run_date variable. The task parameters are defined as follows:

restful

{

"executionContent":{"code":"select \"${f-1}\";","runType":"sql"},

"params":{

"variable":{f:"20.1"},

"configuration":{

"runtime":{

"linkis.openlookeng.url":"http://127.0.0.1:9090"

}

}

},

"source":{"scriptPath":"file:///mnt/bdp/hadoop/1.sql"},

"labels":{

"engineType":"spark-2.4.3",

"userCreator":"hadoop-IDE"

}

}

java SDK

JobSubmitAction.builder

.addExecuteCode(code)

.setStartupParams(startupMap)

.setUser(user)//submit user

.addExecuteUser(user)//execute user

.setLabels(labels)

.setVariableMap(varMap)//setVar

.build

4. Finally, the community is currently holding an essay contest, and the highest reward can be "1000 yuan JD card + 500 yuan community gift", click https://mp.weixin.qq.com/s/KnZutuUfdQnH8XpcmAUNHA to understand