MOSN multi-protocol extension development practice

Service Mesh is a key part of today's cloud native. Ant has completed a large-scale implementation in the production environment, but the overall service mesh transformation in the industry is not high. Among them, the smooth mesh transformation is the prerequisite for the mesh transformation of the online business. In the smooth transformation process, the support of the agreement is the most basic part. The multi-protocol extension development framework provided by MOSN aims to reduce the cost of mesh transformation in scenarios where private protocols are used, and help businesses quickly land.

MOSN is a high-performance network agent self-developed by Ant, which is mainly used for the sidecar of the data plane of Service Mesh. Service Mesh is a hot topic in the cloud-native direction in recent years. Its main purpose is to build an infrastructure layer to be responsible for the communication between services. The main thing is to have a Sidecar deployed together with service applications to realize the basic capabilities of various middleware, realize the standardization of infrastructure, and decoupling from business logic, so as to achieve rapid evolution of business unaware basic capabilities. At present, many domestic companies have begun to embrace Service Mesh, and some companies that cooperate with Ant, such as China CITIC Bank and Jiangxi Rural Credit Co., have also completed the Mesh transformation based on MOSN.

The purpose of the Service Mesh architecture is to reduce the impact of infrastructure transformation and upgrading on the business, but how to smoothly switch from the traditional microservice architecture to the Service Mesh architecture is also a very challenging task. There are many details involved here, but there is one in any case. The most basic problem is that when we are performing gray-scale Mesh transformation, meshed nodes need to be able to maintain normal communication with nodes that are not meshed. The communication protocols chosen by different companies are different, which directly leads to the fact that the selected Sidecar needs to be able to support the protocol used when selecting the technology. Some widely used protocols may still be supported one after another, and some protocols may be customized by the company itself. Therefore, it is inevitable that sidecar-based expansion capabilities and secondary development are required to support private protocols.

Multi-protocol extension framework

When it comes to Sidecar of Service Mesh, you have to mention Envoy, which is a widely used Service Mesh Sidecar proxy. Envoy's extension framework supports developers for secondary development and extension. Many of the protocols currently supported by Envoy are developed and implemented based on its extension framework. Under Envoy's extension framework, to extend a protocol, you can refer to the process of HTTP protocol processing in Envoy, including the 4-layer Filter to implement the codec part and the Connection Manager part, and implement the 7-layer Filter on the basis of the Connection Manager to support additional Business expansion, routing capabilities, and Upstream connection pool capabilities. It can be seen that the process of a protocol processing almost runs through various modules, and the cost of implementing a protocol extension is relatively high.

Let's look at the framework of MOSN again. MOSN can be divided into four levels in a protocol processing. In addition to the basic network layer that obtains data from network IO, it can also be divided into protocol layer, stream layer and proxy layer. Among them, the protocol layer is responsible for the work of protocol parsing related codec, and is responsible for parsing the data stream into protocol frames that MOSN can understand, or encoding protocol frames into binary streams; the stream layer is responsible for more content, including processing different request types , The context of the initial request, the associated event, the association between the response and the request, and the processing related to the upstream connection pool. The details of the processing of different protocols will be different; the proxy layer is a protocol-independent proxy forwarding layer, Responsible for request routing and balancing capabilities, as well as seven-layer expansion capabilities for the expansion of different business implementations. According to this architecture, it can be seen that the core of protocol processing lies in the protocol layer and the stream layer. Compared with the design of Envoy, the routing and seven-layer extension are capable of multiple protocol reuse. But at the same time, it can be seen that the stream layer involves more details and is difficult to implement. For this reason, MOSN has proposed a multi-protocol extension framework on this basis to simplify the implementation of the protocol.

MOSN's multi-protocol framework is mainly aimed at the reuse and extension of the stream layer. In MOSN's protocol processing hierarchical design, the network layer and the proxy layer are designed to be protocol-independent and reusable. If the stream layer can be implemented, the network layer and the proxy layer are reusable. Reuse, then the implementation of the protocol only needs to pay attention to the encoding and decoding of the protocol layer itself, and the difficulty of implementation will be greatly reduced. So does the stream layer have the ability to be reusable? We think it is possible for most protocols, especially the RPC protocol. First, we abstract the protocol and define it as the XProtocol interface, which represents any protocol. Each protocol implementation is to implement an XProtocol interface, which includes the basic codec interface, some special request response construction interface (such as heartbeat, exception), and protocol models (such as HTTP-like pingpong model, common RPC multiple Multiplexing model, etc.), and the interface for protocol matching. All XProtocol protocol implementations are linked through XProtocol Engine. In addition to specifying which protocol to use for processing through configuration, for protocols that implement protocol matching interfaces, they can be automatically identified based on request characteristics. Then we also make a unified abstraction for the protocol frames parsed by XProtocol, including multiplexing-related interfaces and the judgment of the protocol type (whether it is a request or a response, or a control frame like Goaway. The request can be subdivided. For heartbeat requests, unresponsive oneway requests, etc.), support for modification of protocol frame data (Header/Body modification), and unified status code management mapping.

Under MOSN's protocol processing layering mechanism, and with the multi-protocol extension framework with the abstract definition of XProtocol and XFrame as the core, we can process the protocol completely based on the interface at the stream layer, and different protocol extension implementers Only need to focus on the protocol encoding and decoding itself, as well as simple interface adaptation to the results of the encoding and decoding, you can complete the access in MOSN, thereby obtaining the support of various general capabilities in MOSN, such as current limiting extension, Route drainage, etc. Compared with the implementation of the extended protocol in Envoy, it can be seen that it has been simplified a lot. Of course, the MOSN multi-protocol framework cannot meet all protocol conditions, but most of the RPC protocols we have seen so far can be well satisfied with the extension of the seven-layer stream filter in the proxy layer.

Practical case

Let's take the case of MOSN's Dubbo protocol landing in community partners to learn more about MOSN's multi-protocol extension. Many of the codes here are also contributed by students in the MOSN community.
In this case, in addition to the requirement that the protocol needs to support Dubbo, it also hopes to use basic expansion capabilities such as current limiting. At the same time, it needs routing capabilities such as blue and green packets. The selected control plane is Istio for the delivery of dynamic configuration. . So how are these requirements realized in MOSN?

The first is the protocol analysis part. Here we use the open source dubbo-go framework for protocol implementation. Based on dubbo-go, MOSN’s XProtocol and XFrame models are encapsulated; current limiting, xDS and other capabilities directly reuse MOSN’s existing implementations without Additional realization. But there is a problem here is that the routing configuration dynamically issued by Istio is HTTP-related. There is still a certain difference between the HTTP routing configuration model and Dubbo, and the cost of modifying Istio will be higher. Here is another layer of extension. . Based on the MOSN seven-layer StreamFilters, the Dubbo protocol is customized before routing matching to meet the HTTP routing format. There are two main points, one is the Host of HTTP, which corresponds to the Host using Dubbo's Service, and the other is the Path of HTTP. In this part, a default Path is directly added for wildcarding; at the same time, in this extended Filter, it will also get The labels of the machine are added to the Header to match the blue-green grouping function of the route. With the cooperation of MOSN Filter extension, we have realized the standard Dubbo protocol support and satisfied the function of using HTTP routing configuration to meet Dubbo routing.

Plug-in extension

Through the introduction of the MOSN multi-protocol framework, we can understand that when a scene needs to access a custom protocol that MOSN does not support, it needs to be extended and implemented, and then compiled with the framework code of MOSN to obtain a custom protocol that supports MOSN. Although the complexity of the protocol extension implementation has been simplified as much as possible, there are still some problems. For example, in our commercial version of the code, different partners correspond to different scenarios and use different protocols. Then, as the business changes With development, there will be more and more codes related to this part of the agreement, and the corresponding development and maintenance costs will also increase. It’s okay to say, there are still some scenarios, some details of the customer agreement may not want to submit the relevant implementation code to the MOSN warehouse for some requirements, and the commercial version code is different from the open source code, and the source code may not be directly available. Give it to the customer, then there will be problems with the compilation. In order to solve this contradiction, and to further simplify the protocol extension, MOSN has also developed the ability to extend the protocol based on the plug-in model.

The MOSN plug-in extension mode architecture is shown in the figure. MOSN provides a unified plug-in extension framework capability. After MOSN is independently compiled into binary, it uses the plug-in mechanism to dynamically load different protocol extensions to obtain the corresponding protocol support capabilities. In this way, the implementation of protocol extensions can also be independently maintained in the form of plug-ins, and even further may support non-Go language extensions, such as protocol extensions based on WASM extension capabilities to dock with other languages. There are two modes for MOSN's plug-in expansion capabilities, one is based on the Go Plugin mechanism, and the other is based on WASM.

First look at the mechanism of Go Plugin. This is a kind of SO loading capability provided by the Go language. We can compile the code written in Go into SO, which can then be loaded by other Go files. But this mechanism has some big limitations. The first point is that the Go environment compiled by the main program and the extension plug-in must be the same, including the Go version, GoPath and other environment variables; the other point is that the main program and the extension plug-in depend on the library must be the same. This is accurate to Hash, which means that it is not necessary that the two library interfaces are compatible, but must be exactly the same. This restriction is relatively large. Because we expect that the MOSN code and the protocol extension code are maintained independently of each other, the protocol extension code needs to rely on the MOSN framework. According to the Go Plugin mechanism, every time the MOSN framework code changes, the code of the plug-in will be required to be updated simultaneously and then the plug-in is recompiled. , This is too much trouble.

To this end, we disassembled the MOSN framework and split out some relatively stable interfaces and general capabilities as the basis for the common dependence of the MOSN main program and protocol extensions. For example, the interface definitions related to XProtocol and XFrame are individually defined in the API library. Then, when the plug-in is loaded, you only need to register some of the corresponding interfaces in the MOSN framework. Due to relatively few changes in API definitions and tools, protocol extension plug-ins rely on the MOSN framework to change from the MOSN API definition, which minimizes the need to update the plug-in code caused by the MOSN framework code update. For the compilation environment, this is easy to handle. We provide a unified docker environment for compilation. You only need to make MOSN and plugins based on the same docker compilation. The plug-in extension implemented by Go Plugin is the same as the result of directly merging and compiling the code, except that the code of the protocol extension can be maintained independently.

Let's take a look at the extension mechanism of WASM. A brief introduction to WASM, it is a development, efficient, safe, and has the ability to expand the community's unified standards, WASM is theoretically a language-independent ability, and has a widely recognized ABI specification for network agent scenarios. Because MOSN also supports WASM's expansion capabilities, it can also be used for MOSN's protocol expansion.

For detailed introduction of MOSN's WASM extension, please refer to [Practice of WebAssembly in MOSN-Basic Framework]

The protocol extension method based on WASM is different from the protocol extension mentioned earlier. What the WASM plug-in needs to implement is the proxy-wasm ABI standard, without worrying about the MOSN multi-protocol framework, which means that this WASM plug-in can theoretically be used for other applications that follow the proxy-wasm ABI specification. On the MOSN side, a glue layer called WASM Protocol is implemented. This glue layer implements the encapsulation of the MOSN multi-protocol framework, and then interacts with the WASM plug-in through the WASM ABI. From the perspective of the MOSN multi-protocol framework, what we see is an XProtocol implementation encapsulated by the WASM Protocol, and the multi-protocol framework does not understand the interaction part of the WASM ABI. Unlike GoPlugin extensions, MOSN’s WASM framework is still in its infancy, and WASM-based protocol extensions are only POC. For reasons of stability, performance and other factors, it has not been formally applied in the production environment. Need more optimization and testing.

Outlook

Finally, talk about MOSN’s future prospects and plans for protocol support, mainly including more types of protocol support, such as the existence of streaming and duplex gRPC protocols, message-type protocols such as Kafka, MQ, etc. One is to extend the WASM protocol to the production environment.

Recommended reading this week

• MOSN sub-project Layotto: open service grid + new chapter in application runtime
• Development Wasm Protocol Plug-in Guide
• Protocol Extension Base On Wasm-Protocol Extension
• WebAssembly in MOSN-Basic Framework

For more articles, please scan the QR code to follow the "Financial Grade Distributed Architecture" public account