Xu Shiwei: The evolution of Go+

On July 10, the annual ECUG Con 2022 was successfully held online. Xu Shiwei, as the CEO of Qiniuyun, the initiator of the ECUG community, and the inventor of the Go+ language, brought you a keynote speech on "The Evolution of Go+". The following content is based on the speech transcript.

Hello everyone and welcome to ECUG Con 2022. The ECUG conference started in 2007, and it is now in its 15th year, and I bring you speeches almost every year. Following the last conference, this year I would like to continue to share Go+ related content with you and talk about the evolution of Go+. We'll talk about what happened to Go+ in the past? what are we doing now? And how will we continue to iterate in the future?

1. Key Nodes in the History of Go+ Looking at the development of Go+, we will probably divide it into four key nodes.

The first is the v0.5 version and the previous "prehistoric version". Because it was called qlang at the time, it had nothing to do with Go+, so it was called the "prehistoric version". We have now moved the qlang code from Go+ to my personal GitHub. Then there are the prototype versions from v0.6 to v0.7, mainly to let everyone see what Go+ looks like, because it is very different from the previous qlang, qlang is a scripting language, and Go+ is actually a static type of language. After that, we started with this prototype version to bring it closer to engineering use. The more important milestone was last year's v1.0 release, where Go+'s goals and code style were formalized. After that, we basically continued that goal and its code style and moved on. In the first half of this year, we released version 1.1 of Go+, which is actually the first engineered version of Go+, which can be officially used in production environments.

Starting with Go+ 1.0, we first came up with the concept of the "Trinity", oriented towards engineering, STEM education and data science. In fact, we are talking about programming for all, that is, everyone can learn to program. Today, we can see that in the future, more and more people will regard programming education as a basic subject, which is not fundamentally different from mathematics, Chinese, and English. This is why Go+ takes STEM education as a very important support point. So what did Go+ 1.0 do? The first is to determine the code style of Go+, which is based on the command line style, and strives to achieve a low threshold as much as possible. We hope that children aged 7 to 8 will be able to learn Go+. Another very important point is that we implemented the class file beta version. It actually tries to realize the low threshold of object-oriented and domain knowledge expression, that is, the low-code field that is relatively popular now. In fact, although object-oriented is a good thing, it helps to abstract the world, but it also brings difficulties in understanding. Therefore, how to reduce the threshold of these high-level engineering concepts, Go+ class files are the most important exploration in this regard. In addition, Go+ 1.0 made a breakthrough in compatibility with Go syntax, which is why it was finally marked as 1.0. In this version, we have achieved good compatibility with most of the Go syntax, and basically achieved the bottom line goal of extending on the basis of Go. When it comes to the goal of Go+, you may have a lot of questions. In fact, it can be seen from the goal just now that we are very concerned about lowering the threshold. When it comes to low threshold, we have to mention the language Python. What does the success of Python tell people?

First of all, the first important point is that it tells us that performance is not the most important thing. Although everyone pays more attention to performance, in terms of performance alone, I think Python can only be considered second-rate in the scripting language, and it is not fast. Because performance can actually be solved by time. The life cycle of languages is very long. Python has a history of 32 years this year, and its performance problems have the opportunity to be solved iteratively with time. But the characteristics of the language do not, and every choice of language characteristics is a burden in the future. So from this point of view, I also hope that everyone treats language and tries to avoid mediocrity. The second point is that it reveals what is most important to language? Or why does Python succeed? In fact, I think that the choice of target audience is very root cause. The characteristics of language are related to the choice of target group, so the choice of language characteristics is the most important. Since its inception, Python has not positioned itself as a data science language. It believes that the language should be as concise as possible, easy to understand and learn. It is actually a rare low-threshold language, because in my opinion, there are not many languages that can really be called low-threshold. Let's think about a well-known language, such as Ruby, which everyone says is very concise, but in fact it has a lot of language magic. So it might be powerful, but it can't be called easy to learn. So in my mind, BASIC, a language with a low threshold, counts as one, and Scratch, which is oriented to the teaching field, is one. In other words, there are not so many languages for exploration in the low-threshold field, but it is precisely because Python is oriented towards Low threshold, so although it does not position itself as a data science language, it has become the king of data science. This is actually quite ironic, because there are so many languages that focus on data science that are not as successful as Python, and I think there is a very deep truth behind it. From the perspective of the general trend of national programming, in fact, low threshold is the development trend of the mainstream language in the future. Python just conforms to this general trend, making it more successful today than we saw at first. So what's wrong with Python?

First of all, from an engineering point of view, Python is weak in this regard. Talking about Python, some people will think of Go, because there are many programmers who have switched from Python to Go. The reason is that Go is the engineering language with the design idea closest to Python, and the mental burden of users is also very low. But the designers of the Go language basically only have engineering in mind. As you can see from Go's official website, the word it cares most about is scale. That is, how to implement a large-scale project, a project with a very large amount of code, but still under the control of the engineer. So it doesn't care about the low threshold, it cares about how to achieve engineering scale. So from these two aspects, since Go+ is programming for the whole people, then we will naturally pay attention to how to integrate the advantages of Go and Python, and combine the engineering capabilities of Go with the low threshold of Python. We know that engineering is huge, and we are also concerned about implementing larger and more complex systems, but whether it is STEM education or data science, we are more concerned about how to achieve a low threshold. Therefore, the trinity of engineering, STEM education, and data science is actually a fusion of engineering and low thresholds. This is the goal of Go+.

What they look like after merging, we can understand through the following Go+ code example. You may think of Shell programming in the following example, and many programmers may think that Shell scripting is relatively low threshold. Although it is difficult to use in implementing complex tasks, it is the most familiar to everyone in terms of understanding. We will see that the syntax of Go+ is very close to that of Shell.

Let's look at another example, which is to use Go+ to make games. The picture below is actually a dialogue between two characters, a very simple game, and its code is also very concise. Here we will see the shadow of the command line, such as the onStart statement, which is what we should do when the program starts. onMsg is what I do when I receive a message. Basically there are event mechanisms such as onStart and onMsg, and we see that the code inside has say and broadcast (broadcast message). The flow of the entire program is actually composed of several very basic elements through events plus say and broadcast messages.

We can see that this is a dialogue between the two characters. The first character, at the beginning of the program, says where you are from, and immediately broadcasts message 1. When another character receives message 1, he will say I'm from the UK. Then he broadcasts message 2 again. As soon as the character receives Message 2, he will say what the weather is like in your country. In this way, the entire sequence is driven by the message, and the dialogue between the two characters is formed. This program is very concise, through which we can see that Go+ has a natural advantage in expressing natural semantics, and its code is very easy to understand. From these two examples, we can also see that although Go+ is actually a compatible product of Go, its syntax or suggested best practice style is very different from Go. It's even cleaner than Python because of the command-line style of choice. Here we can understand from several concepts of engineering. One is the command, which is an abstraction of a piece of code. In the earliest languages such as FORTRAN, its commands are actually separate from functions. Of course, all later high-level languages basically combine commands and functions. But in Go+, commands and functions are different in code style, but they are all functions behind them.

So in the code style of Go+, we chose the command style as the main body. Because the difficulty of understanding the command is the lowest, elementary school students can understand it. The second is the function, which is basically the concept of junior high school students, such as trigonometric functions. It is not too difficult for junior high school students to understand by combining the functions in computer and mathematics to confirm the two. Basically, you only need to understand the concept of formal parameters and actual parameters. In object-oriented, the concepts of classes and methods, although it does help to abstract the world, in fact, the threshold of understanding object-oriented programming is the highest. So Go+ is actually trying to avoid letting programmers use object-oriented writing. In fact, we will use some object-oriented ideas behind the scenes, but in terms of language syntax, we try to avoid being too object-oriented. So in Go+ 1.0 we saw that it basically laid the Go+ code style and goals.

Now that the goal and style are set, basically the first thing we want is to be able to implement the project and become the first version that can be used in a real production environment. To achieve this goal, two things are most important in terms of priority. One is the support for modules. Everyone knows that Go supports Modules very late, and Go+ is basically compatible with Go's module concept in version 1.1. We have implemented a relatively complete support for modules, which is very consistent with the experience of using Go. Another very important feature is that we implement a hybrid project of Go and Go+. This is very helpful for production environments. Because the first question most programmers face is what to do with Go+? The historical project may be written in Go, so how to convert it into Go+? In fact, you don't need to switch, because your Go project is a Go+ project, and you only need to write some Go+ functions on it. This way, we can use Go+ in production very easily. The next step is to provide a preview of c2go, which is for subsequent releases, and it's a very tough nut to crack. We also basically implemented it in Go+ 1.1. Go+ supports C, and is actually a reference to the c2go project. We have implemented the most basic capabilities of c2go in this version.

Let's look at the version evolution of Go+. If Go+ version 1.0 is a clear goal and a style, then version 1.1 is for the production environment. Whether it is modules or Go/Go+ hybrid programming, they are actually laying the foundation for entering the production environment.

Go/Go+ hybrid programming, as I mentioned just now, any Go project, as long as you add a few Go+ source codes to it, and then replace the go command with gop, you can do Go+ development normally. This actually reduces the barrier to entry for Go+ to a minimum. 2. Go+ current node: v1.2.x

The above is the past of Go+, and now I want to share with you what Go+ is currently doing, which is the Go+ v1.2 version. This version I define it as the process of Go+ feature formation. We expect the official release of Go+ v1.2 in December of this year. We say that this version is a process of characteristic formation, mainly for several reasons. The first is that the class files we introduced in version 1.0 will be normalized, ending the Beta process. Class files are a very important concept in Go+. The second is c2go, which plays a crucial role in the subsequent development of Go+. We hope that the v1.2 version of Go+ will allow c2go to enter the engineering, and its implementation sign is that the migration of sqlite3 is completed at least. In addition to these two very special features, Go/Go+ hybrid programming will also be enhanced. Currently, Go/Go+ hybrid programming does not yet support calling Go generics. We know that the v1.18 version of Go introduced a very important feature that is generics, but now Go+ does not support Go generics. Then we will also support it in the v1.2 version. Instead of defining generics in Go+, we call generics in Go. In this way, we allow Go+ to have the ability to minimize the concept of generics, because generics are a relatively complex concept, but we don't want Go+ to become particularly complicated. Go+ is actually pretty open about generics in the long run, and maybe one day full support for generics will come, but we don't see it as a high-priority thing. If we really need to use generics, we hope to achieve it through mixed engineering with Go. We next focus on these two features, class files and c2go. Let's talk about class files first. The most straightforward explanation of class files is that we use a file to define a class. The right side of the figure below is the method of writing classes in Go, which must be familiar to everyone.

We first define a structure called Rect, which has two members, length and width. Then we define the member method of area, which is the product of length and height. This is a very simple program. If a class file is used to implement this capability, the code can be seen on the left side of the above figure. We basically don't see any object-oriented hiding. We define two global variables, one called width and one called height, and then define a global method called area, which is the product of these two global variables. That code looks much cleaner than normal object-oriented code and is very easy to understand. But in fact these two files are equivalent. Because the most direct ability of a class file is to automatically turn what looks like a procedure-oriented code into an object-oriented method. Because it does not introduce any new grammar, it is actually relatively easy for elementary and middle school students, without having to learn new knowledge.

But class files do more than that. In fact, it can also customize the base class and customize the framework of the entire program execution. Our recent Go+ official account also focuses on class files, you can take a look. Go+ Class File: DSL vs. SDF Go+ Class File: Detailed Explanation of the ClassFile Mechanism

When it comes to class files, it is necessary to mention a design philosophy of Go+: Go+ does not support domain-specific languages, which means that it does not support DSLs, but Go+ is Specific Domain Friendly. Why is it said to be professional field friendly? From our examples above, the first example is Shell programming, or domain programming called DevOps. We can see that the Go+ code looks very close to Shell programming.

In fact Shell programming is more like a DevOps DSL, we rarely use this language outside of Shell, it is only used for very basic automation (automation). But we can see that Go+ can actually make the code of the language itself very close to a DSL, but in fact it is not a DSL, it is a very authentic Go+ syntax. In the same way, we can see that Go+ is also very concise in more complex game programming. And it is not only simple, but more importantly, it is also very powerful, and it can do more complex games like "Plants vs. Zombies". Of course we didn't do a 3D game engine. If we did, it would still be a very streamlined 3D game engine.

It is precisely because Go+ introduces class files that its syntax looks very domain-specific. This means that Go+ is very good at combining domain features to distill domain knowledge. This makes Go+ beneficial to the development of the field. Of course, our class files are still in the Beta stage, and we have already done some practice in the fields of 2D games and DevOps. But after all, there are many fields. For the concept of class files, the biggest challenge is that there are many fields. It is not enough to experiment in a limited number of fields. More fields are needed to verify the universality of the class file mechanism. sex, to judge whether it can adapt to various fields.

In addition, we also need to clear some unnecessary constraints in the beta version class files. For example, our current professional field is only allowed to have one type of work, and it is obviously very likely that this constraint needs to be broken in some professional fields. So we will remove unnecessary constraints like this in the v1.2 version. We want to be able to have a variety of job categories in a professional field, so that it will be more universal. The second feature of the v1.2 version is the c2go just mentioned. The syntax of c2go may look a bit like cgo, but it is completely different from cgo. People complain a lot about using cgo, but using c2go will feel very cool, because we have basically achieved a seamless connection to the C language.

First of all, the code in C language can be called directly by Go+ without additional packaging. Second, we make the type systems of C and Go+ as consistent as possible, which greatly reduces the cost of interfacing between C and Go+. In this case, the two operate with each other, and there is basically no need to convert the type. For example, void in C language is a function with no return value and no parameters. In Go+, it is basically a func(). Such a type of mapping is actually impossible in cgo. Because cgo needs to do the necessary function calling convention conversion, in order to achieve such a call. Finally, after we have translated C, its data structure memory layout and program semantics remain as unchanged as possible. That is to say, C programmers, the general semantic understanding of C code is still correct. For example, a string, which ends with '\x00', is 0. After these concepts are translated by Go+, it is still correct. This will also help people not to disagree on semantics. From the example in the figure below, we can see that through the way of c2go, we have realized the concise call of C.

We can see that the first sentence is import C, but its semantics are completely different from Go. In Go+, it is actually an abbreviation of C/github.com/goplus/libc. Then we can see that in this example we call two C functions: printf and fprintf, using a C variable stderr. Another interesting place is the string. We see that writing a C prefix in front of the standard string in Go+ represents the string constant passed into C in Go+, which is actually a Go+ syntax. But with this syntax, the code for calling C in Go+ will be very streamlined, and it will not be like cgo, there will be a process of converting a Go string to a C string, and finally releasing it. This is a very small example, but we can see that the expression of c2go can make everyone feel as if there is no difference between C and Go packages. The package we introduced C and the package introduced Go are basically the same. And in all the details, it will make everyone feel that the module of C is the module of Go, of course, it is also the module of Go+, which is the final effect we hope to achieve. This is also a logic of our seamless compatibility with C in Go+. Of course, the current c2go is still a preview version, it is not even a beta version. At present, c2go has completed more than 99% compatibility of C syntax. The unfinished part is mainly the migration of the standard C library. Its completion degree may only be 5%, which is in a very early stage. For c2go, its main challenge is cross-platform first. On the one hand, it is the cross-platform of the C standard library (libc), and on the other hand, how to make c2go easily realize the cross-platform capability for all C projects is also a very important capacity building work.

From libc itself, in fact, whether it is syscall, pthread, there is a relatively large workload. syscall Now we have supported the mac version, but Linux and Windows have not yet supported it, not to mention pthread, which is the next section where we will work the most. So the next most important thing for c2go is the migration of the entire standard C library, which itself is a relatively huge work. The above is what the current Go+ v1.2 version wants to solve, and of course there are small details of supporting Go template calls, we will not expand. Basically the above points constitute the characteristic capabilities of Go+, whether it is class files, compatibility with C, and mixed projects of Go and Go+, all of which make Go+ have a very good foundation. 3. Go+ future planning So from the future planning of Go+, everyone knows that Go+ is talking about engineering, STEM education, and data integration. In fact, until v1.2, Go+ has done relatively little in the field of data science. There will be some very limited capabilities to implement, such as list comprehension, range expressions, etc., but it is not systematic. In fact, the data science technology stack of Go+ still does not exist. Therefore, in the very important version of v1.7, we hope that Go+'s own data science technology stack can be formed. Then we made a major version leapfrog to v2.0. We hope that at this stage of v2.0, it will be able to support Python syntax. Of course, it's not about writing Python in Go+. In fact, similar to supporting C, it enables Go+ to import Python packages seamlessly, so that the accumulation of Python data science history can be seamlessly transformed into Go+'s data science capabilities.

In fact, both of these contents are oriented to data science. The reason is that, from the general direction of engineering and low threshold, the v1.2 version is basically complete. Go+ basically doesn't want to spend a lot of effort on syntax. Go+ and Go have very similar philosophies. We believe that the less the syntax, the better, not the more the better. So, basically after the v1.2 version, Go+'s grammar is more stereotyped, and we don't add all kinds of weird grammars. But data science is Go+'s last tough battle, and it's not a simple problem that can be solved in terms of syntax innovation. Go's data science basic capabilities are relatively thin, of course, because Go itself has a relatively short rise time, so it has mostly engineering practices on the server side. So what should I do if the data science foundation is so thin? After the v1.2 version de-engineered the c2go capabilities, it laid an important foundation for the final data science battle. Because the foundation of Python is C, if we are compatible with C, it will be easier to be compatible with Python.

What will we focus on in v1.7? Its goal is actually the formation of the Go+ data science technology stack, and to build Go+'s own data science capabilities, such as the exploration of vectors, matrices and other aspects. In fact, even with such basic capabilities, it is still relatively thin.

How to really solve this problem and put yourself on the shoulders of giants? Our vision is to support Python's data science base (that is, the C library part) through c2go. We hope to be able to make Python's C library part compatible with the v1.7 version, so as to embark on the ecological integration of Go+ and Python data science capabilities. the way. But at this stage, we don't think much about Python itself, and mainly focus on the C library part of Python.

But in the v2.0 version, we began to consider introducing Python's syntax, so that Python's packages would naturally become part of the Go+ ecosystem. We envisage that in the v2.0 version, at least three projects of CPython, NumPy, and pandas should be supported. The first is Python itself, and the other two are the most well-known Python data science projects. With them, we believe that Go+ has the basic ecological capabilities for data science. Here is the code of these three projects, and the structure of the code line is listed. Let's look at Python itself. Most of the base code is C, but 65% of Python's code is actually some standard library, which is easy to understand. But the most core capabilities are completed by C, accounting for more than 30%.

The second is NumPy. NumPy actually does the same thing. Its core capability is written in C, and there is also a small amount of C++. But a toolkit is superimposed on this basis, which is written in Python. As you can see from this line of code, the ratio is basically 6:3 and 6:4.

The structure of pandas is very different. Most of its code is written in Python, and the C part is relatively small. Basically, the v2.0 version can better support pandas. The v1.7 version basically can't support pandas, but it can already support NumPy.

To summarize the evolution of Go+, our goal is to achieve the trinity of engineering, STEM education, and data science. This is mainly because of our future language, the mainstream trend is programming for the whole people, that is, everyone can learn programming. In fact, this goal is difficult, but it is the mainstream trend of future language development. Few languages are currently working towards this trend, and Go+ can be considered the first.

This year, Go+'s exploration of the integration of engineering and low threshold will come to an end. Starting next year, we will launch the final battle for data science. We all know that when Go+ was born we were talking about data science. But when it comes to actually implementing it, data science is the last point. The reason is that data science is really a difficult thing, and for Go it's farther away. But we basically have the foundation of c2go, and we will find that it will have a very important supporting role for us to do a good job in data science in the future. In the end, I believe that Go+ will be unique with the backing of data science. Its future is worth looking forward to together.