A Preliminary Study on Dynamic Execution of Python Code

As the " dynamic " language, Python loads a piece of code and executes it at runtime, which is definitely much more convenient than the "static language" that needs to be compiled (such as C, Java).

execution way

According to whether to return the result, it can be divided into two types: exec and eval.

exec

exec is responsible for executing string codes, can support multiple lines, can define variables, but cannot return results

def pr(x):
    print('My result: {}'.format(x))


if __name__ == "__main__":
    s = '''
a = 15
b = 3
if a > b:
    pr(a+b)
'''
    exec(s)

execution result
> My result: 18

eval

Eval can return results, but can only execute single-line expressions

def select_max(x, y):
    return x if x > y else y


if __name__ == "__main__":
    a = 3
    b = 5
    c = eval('select_max(a , b)')
    print("c is {}".format(c))

execution result
> c is 5

Runtime environment

As can be seen from the above code example, whether exec or eval, their operating environment is the same as the code where they are called: whether it is a global function or a local variable, as long as it is defined before executing the specified code, you can Use, and the variables defined in exec can also be referenced by the code behind.

If necessary, we can also specify the content of the environment definition when running the dynamic code, thereby adding and shielding some information. exec and evel have more than one parameter. Their second and third parameters can specify the globals and locals environments of the dynamic code respectively.

The so-called globals is the global environment when the code is executed. It can be obtained through the globals() function. The return result is a dict, which lists all global variables and global methods, including modules and methods imported by import; the same goes for locals() Functions can return all local variables and local methods.

When we call dynamic code, if we pass the parameters as follows:

def select_max(x, y):
    return x if x > y else y

c = eval('select_max(3 , 5)', {}, {})

The default globals (second parameter) and locals (third parameter) settings will be overwritten, and only the buildin method can be used. At this time, the above code will report an error-because the select_max method cannot be found. In order to make this method usable, we need to assign a value to one of the dicts:

def select_max(x, y):
    return x if x > y else y

c = eval('select_max(3 , 5)', {'select_max':select_max}, {})

This looks a bit like taking off your pants and farting: It's fine to use it directly, why do you cover it first, and then assign the value again? In fact, it is for safety reasons.

safety

Dynamic code capabilities are usually exposed to the outside of the program, allowing configuration personnel to extend program logic. But if there is no restriction, this ability is also very dangerous: for example, by calling the open method, you can open any file and delete its content.

So a safer way is to disable the built-in functions and only expose methods that allow external calls:

def select_max(x, y):
    return x if x > y else y

c = eval('select_max(3 , 5)', {"__builtins__": {}}, {'select_max':select_max})

Note: There are many articles on the Internet, setting __builtins__ to None, after actual measurement, it is not feasible at least in the Python 3.7 environment, it should be set to an empty dictionary

optimization

Compile

The two methods mentioned above are examples of direct execution of strings. We can definitely think that these strings will be “parsed” by the python runtime before being executed, and the parsing process is very time-consuming. Therefore, in order to optimize efficiency, you can use the compile function first, "compile" it in advance, and turn the string into "code", and the efficiency of each execution will be greatly improved.

def select_max(x, y):
    return x if x > y else y


if __name__ == "__main__":
    exp = compile('select_max(a , b)', '', 'eval')
    for i in range(10):
        a = i
        b = i + 10
        c = eval(exp)
        print("c is {}".format(c))

It can be seen that the string becomes an expression after compile, and then calling the expression repeatedly in the loop will be much more efficient than parsing the string each time. Similarly, the content executed by exec can also be compiled into an expression with compile first.

Note: The second parameter of compile is the file name (you can read the code directly from the file), if not, you can

Compiled environment

Compile and eval/exec can not be called in the same function, so they have different execution environments, but in fact, compile does not check the environment. Variables or methods used in dynamic code may not exist at compile time. . For example, change the above code to the following:

exp = compile('select_max(a , b)', '', 'eval')


def select_max(x, y):
    return x if x > y else y


if __name__ == "__main__":
    for i in range(10):
        a = i
        b = i + 10
        c = eval(exp)
        print("c is {}".format(c))

Before defining the select_max method, compile the expression without affecting the running effect at all:

c is 10
c is 11
c is 12
c is 13
c is 14
c is 15
c is 16
c is 17
c is 18
c is 19