We have made a distributed registry

Opening

This article is based on the sharing summary of the SOFA Meetup Hefei station. It is mainly aimed at the positioning and function introduction of the registration center. Through the analysis of the development history of the ant registration center, we will lead everyone to understand how the ant registration center has evolved step by step to the present The size and characteristics of the.

For more in-depth technical details, everyone is welcome to join the SOFA and SOFARegistry communities to explore the results.

What is the registration center

Service discovery & service registration

In simple terms, the registry is to solve the problem of mutual discovery between services in a distributed scenario.

As shown in the figure below, when service A wants to call service B, it needs to know where the address of B is and how to solve this problem?

Generally speaking, it is divided into two points:

1. Service discovery can have a centralized component or storage, which carries the addresses of all services, and at the same time provides a capability for query and subscription. Service consumers can interact with this centralized storage to obtain services The address list of the provider.

2. Service registration: It is also the centralized component mentioned above, but there are two measures for service information at this time

   1. The service connects to the registry and reports its own services and metadata at the same time (which is also the focus of this article today)
   2. There is a centralized control plane to write user-defined services and IP mappings into the registry, such as AWS CloudMap

Call flow

As shown in the figure above, it is currently a mainstream registry model, both SOFARegistry and Nacos.

1. Service A and Service B report their service information to the registry through SDK or REST
2. When service A needs to call service B, it initiates a request to the registry to pull the service IP list and information related to service B
3. After obtaining the list of service B, you can access service B through its own defined load balancing algorithm

Heartbeat

Heartbeat is the default method used by the registry to solve the unavailability of services and pull out services in time to reduce the impact, as shown in the figure below

1. A node of service B is disconnected or hangs, causing heartbeat timeout; or downtime or disconnection directly causes heartbeat failure
2. The registration center pulls the problematic node from its own storage (the pull is based on the specific implementation: some are deleted directly, and some are marked as unhealthy)
3. Service A receives the notification from the registration center and gets the latest list of service B

DUBBO Registry

Let’s take a look at how the registry is used and the process through the example of DUBBO.

First of all, the configuration of DUBBO in 2.7 and 3.0 is slightly different, but they are all simple and easy to understand, and put them all here.

DUBBO-2.7

DUBBO-3.0

On the RPC client, you only need to configure the address of a registry, which contains the basic three elements

1. protocol (protocol type) such as zookeeper
2. host
3. port

Based on this, the registration process of dubbo is shown in the figure below

1. The producer of the service initiates a registration to the Registry through the DUBBO client
2. The consumer of the service subscribes to information through the DUBBO client (subscribe)
3. The registration center issues a list of services to service consumers by way of notification

The nature of the registry

Through the previous explanations and specific examples of DUBBO components, we can probably summarize the essence of the registry

"Storage" + "Operational Maintenance"

1. On the one hand, the registry needs storage capacity to record service information, such as application lists
2. On the other hand, the registration center needs to provide necessary operation and maintenance means in practice, such as shutting down a certain service flow

Chronicles of the Ant Registry

Prehistoric times

Ant in the prehistoric era is a very old architecture. At that time, all services were deployed on the same physical machine or JVM, and there was no cross-machine call scenario between services, so skip it here.

Hard load era

Later, in order to solve the difficulty of deployment and operation and maintenance caused by the coupling between applications, we split the service. After the split, the service encountered a problem, that is, how to deal with the calling relationship between services. At that time, Ants used two hard loads, F5 or LVS.

Through a simple 4-layer proxy, we can deploy the service behind the proxy, and the service and the service can communicate with each other through the proxy, achieving the purpose of cross-machine call

The first-generation registry - the evolution from hard load to soft load

Through the hard load access method, on the one hand, the problem of calling each other between services is solved, and the deployment architecture is also simple and easy to understand; on the other hand, after the rapid growth of business, it has brought certain problems:

1. Single-point problem (if all calls go to F5, once F5 is down, many services will be unavailable)
2. Capacity problem (the traffic carried by F5 is too high, which will cause a performance bottleneck in itself)

At this time, Ant introduced a product of Alibaba Group called ConfigServer, which was used as a registry. The architecture of this registry is very similar to the architecture mentioned at the beginning. Services can be directly accessed through IP, which reduces the number of conflicts. The strong dependence of load balancing products reduces the single point of risk.

The second generation of registration center-ScaleUp? ScaleOut? It's a problem

However, the problem continues, that is, the registration center itself is a single point, then he will continue to encounter the two problems mentioned above

1. Single point of risk (the registry itself is a stand-alone application)
2. Capacity bottleneck (the number of connections to a single registry and the capacity for storing data are limited)

There are two ways to solve it

1. Scale-up (Taobao): By increasing the configuration of the machine to enhance the capacity and the ability to carry links; at the same time, through the main-standby architecture to ensure availability
2. Scale-out (ant): Through the fragmentation mechanism, data and links are evenly distributed on multiple nodes to achieve horizontal expansion; through backup after fragmentation, high availability is achieved

Ants and Taobao took two different paths, and they also promoted the evolution of an independent ecosystem behind the ants.

Ant’s evolutionary architecture is as follows, resulting in two different application nodes

1. The session node is specially used for anti-link use. It is stateless and can be expanded quickly, and the single machine occupies very little resources.
2. Data node, dedicated to storing data, reduces the storage capacity of a single node through sharding, and controls resource occupation

The fifth-generation registry - the birth of the Meta node

The above architecture is already in line with the current mainstream distributed architecture, but during the operation and maintenance process, a series of problems have arisen, such as

1. All data is distributed, and service discovery between data needs to be given a configuration file at startup, so that it is decoupled from standard operation and maintenance
2. The online and offline of the data node needs to modify the configuration file in time, otherwise the cluster restart will be affected
3. Distributed storage consistency issues, each iterative release needs to lock the paas platform to prevent inconsistencies caused by node changes

For all these problems, we found that we can introduce a Metadata Management Center (Meta) node to solve the problem of data and session management. Data and session can access meta through 4-layer load or 7-layer load.

Comparing the solutions in the industry, there are similar models. For example, the Name Node of HDFS and Kafka depend on ZK, Oceanbase depends on RootServer or the configuration center Apollo depends on Euraka.

The emergence of Meta nodes has eased the bottleneck of manual operation and maintenance of the registry, but still has not solved the problem fundamentally, so where is the problem? See the analysis below for details.

The sixth-generation registration center - a registration center for operation and maintenance

As mentioned above, the emergence of Meta nodes has taken over the problem of service discovery between Data and Session. However, in terms of Cong Yun's untested, there are still many problems that cannot be solved, such as

1. The release of Data node is still a pain point under the premise of large amount of data
2. On the newly added node of the Session node, there may be no traffic for a long time

Wait, for these issues, based on SOFARegistry5.x, we quickly iterated version 6.0, mainly for the registration center for operation and maintenance.

The problem of data node release is a problem of scope in the final analysis. How to control the impact of a single data node’s release or suspension on the data is the source of the problem. Here we have adopted two measures.

1. Improve data storage algorithm (consistent-hash -> hash-slot)
2. Application-level service discovery

The evolution of storage algorithms

Previously, we used the consistent hash algorithm. As shown in the figure below, each node carries a part of the data, through the storage to perform the hash operation, calculate the hash value of the stored content, and then calculate the storage that the hash value falls in which data is responsible for Interval to store data.

When the data node is down or restarted, the next data node receives the data of the down node and the data access support.

With this dependence, the granularity of data migration can only be based on the data stored in a single data node. In the case of a large amount of data (single node 8G), it will have a certain impact on data reconstruction, and continuous downtime in data Under the circumstances, there may be scenarios where data is lost or inconsistent.

The improved algorithm, we refer to the algorithm mechanism of Redis Cluster, and use hash slot for data sharding

In this way, during the data publishing process, the data migration can be controlled in units of slots (a single data node has multiple slots, configurable)

At the same time, in order to solve the scenario of inconsistent data writing during migration or downtime, we have introduced a compensation mechanism for data playback. After data is promoted as the master of the slot, it will actively complete a data comparison with all sessions/ Verification, incremental synchronization of newly added data

Application-level service discovery

Application-level service discovery is to solve the problem of large data storage. Because of space reasons, skip the table here

Open source

SOFARegistry has started the open source process from the early stage of the project. The comparison with the current mainstream registry is as follows

We believe that the first thing the registry needs to solve is the issue of availability. Therefore, in the issue of distributed consistency, we chose the AP model, which is also consistent with mainstream registry, such as Euraka and Nacos.

Secondly, in terms of performance, the long connection-based SOFARegistry has a shorter push delay, which is shorter than the push delay of Nacos1.0 (Nacos1.0 is based on the Long Polling model, and Nacos2.0 also uses the long connection model. )

In terms of protocol, SOFARegistry uses the Ant open source protocol stack: the streaming protocol of the BOLT protocol (similar to HTTP2.0), which is more lightweight, and the full duplex mode of the protocol itself: non-blocking, which greatly improves resource utilization.

Feature	Consul	Zookeeper	Etcd	Eureka	Nacos	SOFARegistry
Service health check	Periodic healthcheck (http/tcp/script/docker)	Regular heartbeat to maintain the session (session) + TTL	Periodic refresh(http)+TTL	Regular heartbeat + TTL; support custom healthCheck	Regularly link heartbeat + broken link	Regularly connect to heartbeat + sensitive to broken links
Multiple data centers	support	-	-	-	support	support
Kv storage service	support	support	support	-	support	support
consistency	raft	ZAB	raft	Final consistency	Eventually unanimous (Registration Center) Raft (Configuration Center)	Final consistency
cap	cp	cp	cp	ap	ap+cp	ap
Use interface (multi-language capability)	Support http and dns	Client	http/grpc	Client/http	Client (multilingual) http	Client (java)
watch support	Full/support long polling	support	Support long polling	Not supported (client fetches regularly)	support	Support (server push)
Safety	acl/https	acl	https support	-	https	acl
spring cloud integration	support	support	support	support	support	support

Compared with the well-known Nacos, we have great advantages in financial level and distributed (storage level). In terms of ease of use and cloud native, we are still catching up.

Welcome to join us

One person can go fast, but a group of people can go further

SOFARegistry is an open source project and an important part of SOFA in the open source community. We hope to use the power of the community to promote the development of SOFARegistry instead of only ant engineers. We also launched two projects this year to support more developers to participate:

1. Trun-Key Project (out of the box plan): https://github.com/sofastack/sofa-registry/projects/5
2. Deep-Dive Project: https://github.com/sofastack/sofa-registry/projects/4

The plan is still in its infancy, and everyone is welcome to join in. It can help us solve an issue or write a document, which can better help the community and help ourselves grow.

Recommended reading this week

• RFC8998+BabaSSL---Let the country secret drive to the farther sea of stars
• still worrying about multi-cluster management? OCM is coming!
• MOSN sub-project Layotto: opening a new chapter in service grid + application runtime
• opens a new chapter in cloud native MOSN-integration of Envoy and GoLang ecology

For more articles, please scan the QR code to follow the "Financial-level Distributed Architecture" public account