Yaegi, let you develop hot-swappable scripts and plugins using standard Go syntax

Lead

As a compiled language, Go is often used to implement the development of background services. Since the original developers of Go are all veteran users of C, Go retains a lot of C programming habits and ideas, which is very attractive to C/C++ and PHP developers. As a feature of a compiled language, it also makes Go perform well in a multi-coroutine environment.

But scripting languages are almost all interpreted languages, so how does Go have to do with scripts? Readers, with this question, "listen" to you in this article~~

This article uses the Creative Commons Attribution-Non-Commercial Use-Same Method Sharing 4.0 International License Agreement for licensing.

What kind of language can be used as a scripting language?

Programmers know that high-level programming languages can be divided into two types from the perspective of operating principles: compiled languages and interpreted languages. Go is a typical compiled language.

• Compiled language requires the use of a compiler to compile the code into a machine code file that can be directly recognized by the operating system before the program runs. When running, the operating system directly pulls up the file and runs directly in the CPU
• Interpreted languages need to pull up an interpreter before the code runs, and use this program to execute according to the logic of the code at runtime.

Typical examples of assembly language, C, C++, Objective-C, Go, Rust, etc.

Typical examples of interpreted languages are JavaScript、PHP、Shell、Python、Lua and so on.

As for Java , from the perspective of the JVM, it is a compiled language, because the compiled binary code can be directly executed on the JVM. But from the perspective of the CPU, it is still an interpreted language, because the CPU does not directly run the code, but indirectly interprets the Java binary code through the JVM to achieve logical operation.

The so-called "scripting language" is another concept, which generally refers to a programming language designed to develop a small program or small logic, and then use a preset interpreter to interpret this code and execute it. This is a functional definition of a programming language. In theory, all interpreted languages can be conveniently used as scripting languages, but in fact we don't do this. For example, PHP and JS are rarely used as scripting languages.

As you can see, interpreted languages are inherently suitable as scripting languages because they inherently need to use runtime to interpret and run code. A slight modification or encapsulation of the runtime can realize a function of dynamically pulling up the script.

However, programmers do not believe in evil, and they have never given up their efforts to turn compiled languages into scripting languages.

Why do I need to write scripts in Go?

First answer a question: Why do we need to embed a scripting language? The answer is simple. The compiled program logic has been fixed. At this time, we need to add an ability to adjust certain parts of the functional logic at runtime to achieve flexible configuration of these functions.

In this regard, the project team actually has relatively mature applications for Go and Lua, using yaegi and gopher . There are already many articles about the latter, so I won't repeat them in this article. Here we briefly list the advantages of yaegi

• Fully follow the official Go grammar ( 1.16 and 1.17 ), so there is no need to learn a new language. However, generics are not currently supported;
• Go native libraries can be called, and third-party libraries can be extended to further simplify the logic;
• The Go program with the struct can directly use 0617a0b3fd3797 for parameter transfer, which greatly simplifies development

It can be seen that in yaegi's three advantages, there are "jian" characters. Easy to use and easy to dock are its biggest advantages.

Get started quickly

Here, we write the simplest piece of code, the function of the code is Fibonacci number:

package plugin

func Fib(n int) int {
    return fib(n, 0, 1)
}

func fib(n, a, b int) int {
    if n == 0 {
        return a
    } else if n == 1 {
        return b
    }
    return fib(n-1, b, a+b)
}

Make the above code a string constant: const src = ... , then use yaegi to encapsulate and call in the code:

package main 

import (
    "fmt"

    "github.com/traefik/yaegi/interp"
    "github.com/traefik/yaegi/stdlib"
)

func main() {
    intp := interp.New(interp.Options{})  // 初始化一个 yaegi 解释器
    intp.Use(stdlib.Symbols)  // 允许脚本调用（几乎）所有的 Go 官方 package 代码

    intp.Eval(src)  // src 就是上面的 Go 代码字符串
    v, _ := intp.Eval("plugin.Fib")
    fu := v.Interface().(func(int) int)

    fmt.Println("Fib(35) =", fu(35))
}

// Output:
// Fib(35) = 9227465

const src = `
package plugin

func Fib(n int) int {
    return fib(n, 0, 1)
}

func fib(n, a, b int) int {
    if n == 0 {
        return a
    } else if n == 1 {
        return b
    }
    return fib(n-1, b, a+b)
}`

We can notice the fu variable, which is directly a function variable. In other words, yaegi directly exposes the functions defined in the script to the main caller program after interpretation as a function of the same structure. The caller can directly call it like a normal function, instead of calling it like other script libraries. A special function of passing parameters, obtaining the return value, and finally converting the return value.

From this point of view, it is very, very friendly, which means that parameters can be passed directly between the runtime and the script without intermediate conversion.

Custom data structure delivery

As mentioned above, a great advantage of yaegi is that it can directly pass custom struct format.

Here, I first throw out the method of how to pass a custom data structure, and then further talk about yaegi's support for third-party libraries.

For example, I defined a custom data structure , and hope to pass it in the Go script:

package slice

// github.com/Andrew-M-C/go.util/slice

// ...

type Route struct {
    XIndexes []int
    YIndexes []int
}

Then, when initializing the yaegi interpreter, we can call the following code to initialize the symbol table after the initialization of the intp variable is completed:

    intp := interp.New(interp.Options{})

    intp.Use(stdlib.Symbols)
    intp.Use(map[string]map[string]reflect.Value{
        "github.com/Andrew-M-C/go.util/slice/slice": {
            "Route": reflect.ValueOf((*slice.Route)(nil)),
        },
    })

In this way, the script when calling, in addition to native libraries, can also be used github.com/Andrew-M-C/go.util/slice in Route structure. This realizes the native transfer of struct.

1617a0b3fd3991 is needed Note that is: Use , whose key is not the name of the package, but the combination of package path + package name. For example, to introduce a package, the path is: github.com/A/B , then its package path is " github.com/A/B ", the package name is B , and the key connected is: github.com/A/B/B . Note that the following " B " is repeated twice. Been pitted by this, stuck for several days.

Yaegi supports third-party libraries

principle

We can pay attention to the intp.Use(stdlib.Symbols) above example, which can be said to be one of the implementations that distinguish yaegi from other Go script libraries. The meaning of this sentence is: use the symbol table of the standard library.

After the Yaegi interpreter analyzes the syntax of the Go script, it will link the symbol calls in it with the targets in the symbol table. And stdlib.Symbols exports almost all the standard library symbols in Go. However, from a security perspective, yaegi prohibits high-privilege system calls such as poweroff and reboot.

Therefore, we can naturally think that we can also define a custom symbol table-this is Use function. By defining the prototype of each symbol to yaegi , the third-party library can be supported.

Of course, this method can only pre-define third-party libraries that the script can reference, and does not support dynamic loading of undefined third-party libraries in the script. Even so, this greatly expands the capabilities of yaegi scripts.

Symbol resolution

In the previous article, we manually specified the third-party symbol table to be imported in the code. But for very long codes, it is too troublesome to type symbol by symbol. In fact, yaegi provides a tool that can analyze the target package and output a list of symbols. We can take a look at yaegi's stdlib library as an example. It explains the Go native package file and finds the symbol table. The package used is a tool developed with yaegi.

Therefore, we can borrow this function and combine with go generate to dynamically generate the symbol table configuration code in the code.

Take the above github.com/Andrew-M-C/go.util/slice as an example. In the place where yaegi is referenced, add the following go generate:

//go:generate go install github.com/traefik/yaegi/cmd/yaegi@v0.10.0
//go:generate yaegi extract github.com/Andrew-M-C/go.util/slice

The tool will generate a github_com-Andrew-M-C-go_util-slice.go file in the current directory. The content of the file is the symbol table configuration. In this way, we don't have to spend time exporting symbols one by one.

Comparison with other scripting solutions

Function comparison

In addition to researching yaegi, we also researched and compared tengo gopher-lua using Lua. The latter is also a more mature library used by the team.

The author needs to emphasize that although the title of tengo says that he uses Go, it is actually selling dog meat. It uses its own set of independent grammar, which is completely incompatible with official Go, and it is not even similar. We should treat it as another scripting language.

The comparison of these three schemes is as follows:

	yaegi	tengo	gopher
Programming language	Go	tengo	Lua
Active community	Within 1 day	Within 1 month	5 months ago	Note: As of October
Complex type	Pass directly	not support	Pass with table
official version	no	Yes	no	Note: Gopher has no official release version, but it is relatively stable
Standard library	Go standard library	tengo standard library	Lua standard library
Tripartite library	Go third party library	without	Lua third party library	Note: yaegi does not currently support cgo
performance	middle	Lower	high	Note: See "Performance Comparison" below

all in all:

• The advantage of gopher lies in performance
• The advantage of yaegi lies in Go's native syntax and acceptable performance
• What is the advantage of tengo? For this author’s use scenario, there is no

But yaegi also has obvious shortcomings:

• It is still at 0.y.z version, which means this is only a beta version, and subsequent APIs may have relatively large changes
• The general direction of Go's official grammar is to support generics, and yaegi currently does not support generics. In the future, we need to pay attention to yaegi's convenient iterative situation

Performance comparison

There are many tables below, here is the comparison conclusion of these three libraries:

• From the perspective of pure computing power performance, gopher has an overwhelming advantage
• The performance of yaegi is very stable, about 1/5 ~ 1/4 of gopher
• In non-computationally intensive scenarios, tengo's performance is worse. The average scene is also the worst

Simple a + b

This is a simple logic package, which is the ordinary res := a + b , which is a test of extreme conditions. The test results are as follows:

Package names	Scripting language	Time consuming per iteration	Memory footprint	alloc number
Go native	Go	1.352 ns	0 B	0
yaegi	Go	687.8 ns	352 B	9
tengo	tengo	19696 ns	90186 B	6
gopher	lua	171.2 ns	40 B	2

The results are surprising. For particularly simple scripts, tengo is extremely time-consuming, and it is likely that too many resources are consumed when entering and exiting the tengo VM.
And gopher showed excellent performance. It is very impressive.

Conditional judgment

The logic is also very simple, judging whether the input number is greater than zero. The test result is similar to simple addition, as follows:

Package names	Scripting language	Time consuming per iteration	Memory footprint	alloc number
Go native	Go	1.250 ns	0 B	0
yaegi	Go	583.1 ns	280 B	7
tengo	tengo	18195 ns	90161 B	3
gopher	Lua	116.2 ns	8 B	1

Fibonacci number

The previous two performance tests are too extreme and can only be used for reference. In tengo's README, it claims that it has very high performance, comparable to gopher and native Go, and can overwhelm yaegi. Now that tengo is so confident and gave out the Fib function it uses, then I will test it. The test results are as follows:

Package names	Scripting language	Time consuming per iteration	Memory footprint	alloc number
Go native	Go	104.6 ns	0 B	0
yaegi	Go	21091 ns	14680 B	321
tengo	tengo	25259 ns	90714 B	73
gopher	Lua	5042 ns	594 B	1

Let's put it this way: tengo claims to be equivalent to native Go, but in fact it is two orders of magnitude worse, and the performance among these competitors is the lowest.

This test result is also very different from the benchmark data declared on tengo's README. If readers know what tengo's test method is, or if there is a problem with my test method, I hope to point it out~~

Points for Attention in Engineering Application

In actual engineering applications, for yaegi, the author locks in such an application scenario: using Go runtime programs to call Go scripts. I need to restrict this script to complete limited functions (such as data inspection, filtering, and cleaning). Therefore, we should limit the ability of the script to be called. We can do this by deleting stdlib.Symbols table. In actual applications, the author has deleted the following package symbols:

• os/xxx
• net/xxx
• log
• io/xxx
• database/xxx
• runtime

In addition, although yaegi directly exposes the script function and can be called directly, the main program cannot make any assumptions about the reliability of the script. In other words, the script may panic, or modify the variables of the main program, causing the main program to panic. To avoid this, we need to run the script in a restricted environment. In addition to restricting the packages that yaegi can call, we should also restrict the way the script is called. Including but not limited to the following methods:

1. Put the calling logic in a separate goroutine to call, and catch the exception recover
2. Do not directly expose the memory information such as the variables of the main program to the script. When passing parameters, you need to consider copying the parameters before passing them, or the possibility of the script returning illegally
3. If not necessary, you can prohibit the script from opening new goroutines. Since go is a keyword, the full text matches the regular " \sgo " (note the space character).
4. The running time of the script also needs to be limited or monitored. If there is an infinite loop in the script with a bug, the caller should be able to break away from this script function and return to the main process.

Of course, the article is full of disrespect for tengo, and it's just that tengo does not have any advantages in the author's use scenario. Readers are invited to read dialectically, and supplements and corrections are also welcome~~

---

This article uses Creative Commons Attribution-Non-Commercial Use-Same Method Sharing 4.0 International License Agreement for licensing.

Original author: amc , the original text was published in Cloud + Community , which is also the blog himself Welcome to reprint, but please indicate the source.

Original title: "Yaegi, let you develop hot-pluggable scripts and plug-ins using standard Go syntax"

Release Date: 2021-10-20

Original link: https://cloud.tencent.com/developer/article/1890816 .