Zhonglun network full station Dubbo 2 migration Dubbo 3 summary

Author: Liu Jun

Zhonglun Network will complete the site-wide upgrade of the service framework from Dubbo 2 to Dubbo 3 in 2022, and deeply utilize core capabilities such as application-level service discovery, Kubernetes native service deployment, and service governance. Lai Binbin, the technical director of Zhonglun Network, made an in-depth summary of the selection, upgrade process and benefits of the entire Dubbo 3.

Lai Binbin, joined Suzhou Zhonglun Network Technology Co., Ltd. in 2020 as the head of the architecture department/senior architect, with 14 years of experience in architecture development. He worked in Suning.com for seven years and participated in the architecture design of the merchant platform and the logistics Quartet service platform, providing users with 100 million-level SaaS services. Now we focus on ensuring the implementation of China-Taiwanization, performance optimization, business architecture, stability, etc., focusing on the retail industry, Internet, cloud computing, and architecture design.

It is worth mentioning that the documents on the official website of Dubbo 3 have been substantially improved recently, and the community promises that a lot of energy will be devoted to improving the documents in the short term, which is very important for the use of Dubbo 3 and the improvement of user confidence.

Introduction to the company's business and technical structure

Suzhou Zhonglun Network Technology Co., Ltd. is a company that "focuses on retail store revenue-increasing services". It has always been "solving the difficult problems of small and medium-sized retail stores" as its original intention, and is committed to providing retail stores with integrated solutions for store operations and helping retail stores. Realize income increase. With retail technology as the core, Zhonglun Network has created a new system for retail merchants that integrates cash register system, Zhonglun shopkeeper, micro mall, Huilin life platform, big data platform, mobile payment, smart farmer, and Huilin store operation services. The retail ecosystem realizes the all-round integration of online and offline, enabling retailers to increase their income. At the beginning of the construction, the technical team selected Dubbo 2.5.3+Zookeeper version to build the company's micro-service base to support the company's business development. Later, in-depth cooperation with Alibaba Cloud, the overall migration and use of Alibaba Cloud, using the cloud native infrastructure ACK (Kubernetes) + MSE (Zookeeper) )+Dubbo+PolarDB and other constructions to realize the service capability of dynamic scaling and expansion.

With the expansion of 3000+ partners, the market covers 300+ cities, 300,000+ retail merchants, and services cover catering, tea, clothing, mother and baby, baking, fresh food, supermarkets, beauty, beauty, pets and other industries . With the expansion of the field and the rapid increase in the number of merchants, the number of systems and deployment nodes have also ushered in a surge, and the system availability has been greatly challenged: microservice governance capabilities, microservice address registration discovery, Kubernetes platform services The lossless There are more and more problems and challenges in the smoothness of online and offline. The architecture diagram is shown in Figure 1.

Figure 1

Dubbo 3 upgrade summary

In upgrading the technology selection of microservice components, the main consideration is to solve the previous pain points: service governance capability, cloud native friendliness, service registration discovery, these are several critical issues that restrict business development. In comparison, the architectural design concept of Dubbo 3 is more in line with ours and can better meet our business development requirements.

1. Service Governance Capability

Dubbo 3 provides rich service governance capabilities, which can realize service governance requirements such as service discovery, load balancing, and traffic scheduling. We have two options to use:

• Manage configurations using the Dubbo Admin Console
• Integrate related API capabilities into the system

At the same time, Dubbo has good scalability, and can customize its own implementation in many function points (see Figure 2) to change the default behavior of the framework to meet its own business needs.

Dubbo SPI (Service Provider Interface) configures the fully qualified name of the interface implementation class in the file, and the service loader reads the configuration file and loads the implementation class. This allows you to dynamically replace the implementation class for the interface at runtime. Based on this feature, we can easily provide extended functions for our programs through the SPI mechanism. For example, we have implemented link monitoring based on production and consumer filters to achieve full-link customization; based on routing extensions The label routing method isolates the test environment, which is convenient for fast multi-version service test verification. In practice, we mark the service based on the producer's registration. For example, the original system A V1 version is deployed in the fat environment. Now in order to test the V2 version, we mark the V2 version with tag=fat-v2; the consumer specifies Invocation when consuming Attachment parameter, inv.setAttachment(TAG_KEY, routeTag); based on this, we can facilitate self-testing, and we can also do simple production grayscale applications in production.

Figure II

2. Cloud-native friendliness

Dubbo follows the cloud-native microservice development concept in design. Microservices support Kubernetes platform scheduling to achieve the alignment of service life cycle and container life cycle, including life cycle events such as Dubbo's startup, destruction, and service registration. Zhonglun network microservice management uses MSE (Zookeeper), so our service exposure needs to be aligned with it. In terms of specific operations, we customize the Startup startup probe, the Liveness survival probe, and the Readiness readiness probe. The normal switching of the project needs to ensure the lossless online and offline. In the implementation, the lossless online will be more troublesome than the offline. The project release and online process will generally follow the following process (roughly divided into three stages):

In the first stage, upgrade a small number (such as 20%) of instances, and switch a small amount of traffic to the new version. After completing this stage, suspend the upgrade first.

After manual confirmation, continue to the second stage, upgrade a larger proportion (such as 90%) of instances and traffic, and pause again for manual confirmation.

The final stage will be fully upgraded to the new version and verified, thus completing the entire release process. If any abnormality including business indicators is found during the upgrade, such as abnormally high CPU or memory usage or too many requests for 500 logs, it can be rolled back quickly.

Because we are using the MSE (Zookeeper) service, the self-registration of the Dubbo service is exposed during the application startup process and is not controlled by the Kubernetes life cycle. There is a problem that some services that are not fully ready for the project can be accessed in advance.

Figure 3

In the implementation process, we mainly use the Dubbo Qos command, and the initial use of the service is not exposed. After the application is ready, the Qos online command is called to replace the old node with the service online. The number of nodes replaced each time is determined based on the release strategy; the offline process is based on the need to be offline. For nodes, we will first use the QoS command to perform offline operations and wait for the application to complete the service, so as to stop gracefully. From the practical effect, it can meet our production needs.

3. Instance level upgrade switch

Compared with the interface granularity-based service discovery mechanism in the 2.x version, 3.x introduces a new application granularity-based service discovery mechanism, which further improves the performance and stability of Dubbo 3 in large-scale cluster practice. During the upgrade process, we also introduced the configuration center and the original data center simultaneously, which will enable the gray part in Figure 4:

Figure 4

Instance-level registration management is adopted, and an application has N services. At the interface level, N services need to monitor the push, and the application level only pays attention to information related to a single instance. At the same time, after the introduction of the metadata center, the interface configuration data information is greatly reduced, the interface data transmission size is reduced, and the configuration of related responsibilities is also clearer. According to the test of the new model, the utilization rate of system resources is greatly improved, the memory consumption of the single machine of the Dubbo address is reduced, the storage and push pressure of the registry cluster is greatly reduced, and the stability is greatly improved after going online.

Summary and Outlook

In the process of upgrading Dubbo 3 from Dubbo 2 of Zhonglun Network, we also had some hesitations, such as changing the interface level to application level or mixed registration; the timing and scope of introducing new features and technologies of Dubbo 3.0 into the project. For the company, a major upgrade means risks and unpredictable problems, but at the same time, it can also bring about the improvement of resource utilization and the expansion and enhancement of basic functions. As technicians, we need to repeatedly evaluate and demonstrate carefully. Now we are done switching all business areas.

Dubbo Community Collaboration

Here again, I would like to thank the Dubbo community staff for their professionalism, efficiency, and strong support for the upgrade of Zhonglun's network architecture. At the same time, I am honored to be a contributor to the community. Interested students can join the contributor DingTalk group: 31982034.

The Dubbo community recently organized a weekly microservice pure technology sharing, explaining the use and source code of Dubbo, and covering a lot of cloud-native microservice knowledge. Interested students can pay attention to the Apache Dubbo official account.

MSE registered configuration 20% off the first purchase, 30% off the first purchase of 1 year and above. 10% discount for MSE cloud native gateway prepaid full specification. Click here to enjoy the discount.