Registration Center of Douyu Live Cloud Native Practice

author

Kong Lingzhen, Chief Architect of Douyu, is fully responsible for the planning and construction of Douyu's entire site technical architecture system. He has more than 10 years of experience in the architecture of medium and large-scale Internet products. He is good at architecture and solution design in high-concurrency and high-availability scenarios.

Yu Jing, Betta technical support operation and maintenance expert, is responsible for the construction of Betta high-availability infrastructure, is good at registration center, monitoring system and other technical fields, and is also the person in charge of Betta multi-life basic guarantee.

Tang Cong, senior engineer of Tencent Cloud, author of the geek time column "etcd Practical Course", active contributor to etcd, mainly responsible for Tencent Cloud's large-scale k8s/etcd platform, stateful service containerization, offline mixing and other product development and design work .

Chen Peng is the product architect of Tencent Cloud Container Service. He has focused on the cloud-native field for many years and has helped a large number of users in the transformation and production of cloud-native containerization. He has rich first-line practical experience and has also published a large number of cloud-native technical articles.

Business background and pain points

As the industry's leading game live broadcast platform, Douyu Live provides high-quality live game viewing, interaction and entertainment services for hundreds of millions of Internet users every day.

With the boom in the live broadcast market in recent years, the Douyu live broadcast platform, as an Internet company with excellent reputation and experience in the industry, has also seen a spurt of user growth. The stability and technical challenges brought by a large number of users to the platform are becoming more and more intense. The old architecture of Douyu is shown in the figure below. Whether it is business support or architecture design, there are certain risks and hidden dangers.

Betta old architecture

Figure 1 The old architecture of Douyu

In order to provide users with a better usability experience, Douyu urgently needs to solve the problem of a single data center and upgrade the old architecture from a single data center to multiple data centers.

Multiple data center challenges

In the process of upgrading from single-active to multiple-active, in order to ensure a trouble-free migration and upgrade, we face a series of challenges, such as:

How to synchronize stateful services etcd, zookeeper, etc. to multiple data centers?
There is a complicated tree-like or net-like dependency between applications. Where should we start to migrate?
According to what dimension to divide the boundary of the target, how to avoid the business welding together, resulting in a situation where there is no way to start?
If there is a problem after the migration, how to recover quickly without affecting the successfully migrated business?
Because there are many systems involved in the process of upgrading from single live to multi live, this article will be the first in the Douyu live broadcast multi live modification series, focusing only on the registry module, so we first introduce you to the etcd and etcd behind the registry zookeeper.

The role played by zk/etcd

dubbo uses the registry to solve the problems of service registration and discovery in large-scale clusters. The following is the architecture diagram of the registry:

dubbo supports the zookeeper registry by default. Although the new version also has etcd implementation, this implementation lacks a precedent for large-scale production. It is also rare for the Java technology stack to use etcd as the registry.

When zookeeper is used as the dubbo registration center, its registration relationship is a tree structure, and the detailed structure is shown in the following figure:

Because zookeeper is based on a tree structure similar to a file system to store data, but etcd uses key-value pair storage, the difference between the two will bring greater difficulties to the registration relationship synchronization.

In addition, if you migrate from zookeeper to etcd, during the entire migration process: existing online services cannot be damaged, let alone stop the service; if the migration fails, you can also fall back to zookeeper.

New architecture of dual-active and multiple-active in the same city

In order to achieve multiple activities, we successfully solved the above challenges through technical means and operation and maintenance concepts such as cross-data center synchronization services, service dependency combing and boundary division, controllable changes, and designed the following set of new architectures to achieve multiple activities ,As shown below:

Figure 2 The new architecture of Douyu Duohuo

Under the new architecture, traffic can be fine-grained by domain name or even URL. The RPC level also has the ability to automatically call nearby. The partial architecture diagram of the registry is as follows:

Figure Three Old Architecture of Douyu Registration Center

Selection and Goal of the Multi-Activity Scheme of the Registration Center

In the process of multi-activity transformation of the registration center, we are faced with multiple solutions, as shown in the following table:

Due to historical reasons, we have two sets of registration centers, zookeeper (hereinafter referred to as zk) and etcd, plus we have four technology stacks: Java, Go, C++, and PHP. Therefore, there are still some deficiencies in the field of registration centers. etcd to solve the pain points and achieve the following goals:

Reduce maintenance costs: Previously, it was necessary to operate and maintain two sets of registration centers, zk+etcd. What is more difficult is that zk+etcd needs to be adapted for multi-activity solutions, which causes the multi-activity R&D cost of the registration center to double. Since etcd is a part of k8s, it is inevitable to operate etcd. This is the first reason for choosing etcd.

Embrace a more prosperous ecology: etcd has cloud native hosting solutions, and some vendors manage 10K node-level k8s clusters through etcd. etcd also comes with various peripheral tools such as proxy, cache, and mirror. Java side dubbo also supports etcd as registration In the center, etcd has better development prospects than zk, which is the second reason for choosing etcd.

Enhanced cross-language capabilities: etcd can communicate based on http or grpc protocol, and supports long polling, with strong cross-language capabilities. And zk needs to introduce a dedicated client, in addition to the java client, other language clients are not yet mature. And we have 4 R&D languages such as JAVA, Go, C++, PHP, etc. This is the third reason for choosing etcd.

Based on the above reasons, we chose option four. The four new architectures of the four options are shown in the figure below:

Figure Four Douyu Registry New Architecture

Difficulties and challenges for the registration center

In order to realize the new registration center and achieve our desired design goals, the registration center faces the following difficulties and challenges during the transformation process:

How to solve zk's multi-data center synchronization problem? Especially the zookeeper watch mechanism is unreliable, maybe there is a problem of missing watch events? (Correctness)
How to solve etcd's multi-data center synchronization problem? From the following scheme selection, we can see that the community currently does not have any mature, production-based solutions available. (Correctness)
How to solve the performance problem of cross-data center reading? (performance)
How to solve the problem of service stability across data centers? What should I do if the network link, such as the private line of the intranet, is interrupted? In terms of the synchronization service design, will the etcd/zk synchronization service enter into the full synchronization logic with extremely slow performance? Does the synchronization service itself have high availability, etc.? In disaster tolerance testing, how should we design test case verification? In operation and maintenance, how do we quickly discover hidden dangers, eliminate potential failures, and build a visual and flexible multi-active operation and maintenance system? (Stability, Operability and Maintenance)

Difficulty analysis of the registration center

How to ensure the interoperability of new and old services during the migration process?

Develop zk2etcd

Many of our java-developed businesses use the dubbo framework for service management, and the registry is zookeeper. We hope that all businesses developed by java and go will use etcd as the registry, and pave the way for the possibility of cross-language calls.

Due to the large number of businesses, the transformation and migration cycle will be very long, and it is expected to last for 1 to 2 years. During this process, we need to synchronize the registration data in zookeeper to etcd for real-time synchronization, and to ensure data consistency and high availability. Currently, there is no tool on the market that meets our needs, so we cooperated with the Tencent Cloud TKE team to develop a zk2etcd to synchronize zookeeper data to etcd, and we have open sourced it. We will introduce the overall solution in detail in the landing article.

How to realize etcd remote disaster recovery?

Through the zk2etcd synchronization service, we have successfully solved the zookeeper data migration problem, so that the registry data of the new and old businesses are stored in etcd.

Therefore, the importance of etcd is self-evident. Its availability determines our overall availability. The current deployment architecture of Douyu Live heavily relies on a core computer room. Once the core computer room fails, it will be unavailable as a whole. Therefore, the next pain point of Douyu Live is to improve the usability of etcd, hoping to realize etcd's cross-city disaster tolerance and remote disaster tolerance capabilities.

The ideal etcd cross-city synchronization service of Douyu Live should have the following characteristics:

After etcd is deployed in cross-city disaster recovery, the read and write performance is not significantly reduced, which can meet the basic requirements of business scenarios.
The synchronization component reaches the usable level of the production environment, with complete consistency detection, logs, metrics monitoring, etc.
Businesses that do not require strong data consistency can access nearby etcd cluster services in the same region, and businesses that require strong consistency can access the main etcd cluster.
After the primary cluster fails, business operation and maintenance can quickly upgrade the standby cluster to the primary cluster based on consistency monitoring, etc.
So what are the options? What are the advantages and disadvantages of each program? Finally, the following options were evaluated:

Single cluster multi-location deployment solution

etcd community make-mirror solution
etcd community learner solution
Tencent Cloud etcd-syncer solution
Single cluster multi-location deployment solution
The single-cluster multi-location deployment plan is as follows:

In this scheme, the etcd Leader node replicates data to Follower nodes in various regions through the Raft protocol.

The advantages of this program are as follows:

After the regional networks are interconnected, the deployment is simple and no additional components are required for operation and maintenance

Strong and consistent data synchronization across cities. In a 3-node deployment scenario, failures in any city can be tolerated without any data loss

After introducing its advantages, let's take a look at its disadvantages, as follows:

In a three-node deployment scenario, any write request requires at least two nodes to respond and confirm, and when different nodes are deployed in various places, the ping delay will increase from a few milliseconds to about 30ms (Shenzhen-Shanghai), which will lead to a sharp drop in write performance .

The default read request of etcd is linear read. When the follower node receives the read request initiated by the client, it also needs to obtain relevant information from the leader node and confirm that the local data catches up with the leader before returning the data to the client to avoid reading old data In this process, etcd read latency will increase and throughput will decrease.

The quality of inter-city deployment networks is also prone to fluctuate, leading to jitters in service quality.

In order to prevent a single point of failure, multiple etcd nodes must be configured for the client to access the etcd cluster configuration, which in turn may cause the client to access the etcd node in a remote place, leading to increased service request delays, etc.

etcd community make-mirror solution

After introducing the single-cluster multi-location deployment solution, let's take a look at the make-mirror solution provided by the etcd community. Its schematic diagram is as follows:

In this solution, we deployed a set of independent etcd clusters in different cities, and implemented cross-city data replication through the make-mirror tool provided by the etcd community.

The principle of the make-mirror tool is as follows:

After specifying the prefix for data synchronization, all data under this prefix is traversed from the main cluster through the etcd Range read interface and written to the destination etcd. (Full synchronization)
Then specify the "version number" returned by the read request through the etcd Watch interface, and monitor all change events from this version number.
After make-mirror receives the key-value change event pushed by the main etcd cluster, it writes the data to the hot standby cluster through the txn transaction interface. (Incremental synchronization)
The advantages of this program are as follows:

The main etcd cluster has high read and write performance, and is not affected by cross-regional network delays and network quality fluctuations as a whole
If the business can tolerate short-term inconsistencies, you can visit the nearest etcd cluster
If the business requirements are strongly consistent, you can access the main etcd cluster through the private line of the intranet
Does not rely on high version etcd
After introducing its advantages, let's take a look at its disadvantages, as follows:

When the write request is large, there may be a certain amount of data lag in the standby cluster, and dirty data may be read.
After the community's built-in make-mirror synchronization link is interrupted, exiting and restarting will enter the full synchronization mode again, with poor performance and unable to meet the requirements of the production environment.
The make-mirror tool that comes with the community lacks a series of features such as leader election, data consistency detection, logs, and metrics, and does not have the usability of the production environment.
It does not support synchronization of non-key-value data, such as auth authentication related data, lease data, etc.

#### etcd community learner program
After introducing the make-mirror solution of the etcd community, let's take a look at the learner solution provided by the etcd community. Its schematic diagram is as follows:

Its core principles are as follows:

The etcd raft algorithm library has already supported the learner node in 2017. For details, please refer to pr 8751.
The etcd community officially supports the Learner node in the 3.4 version launched in August 2019. It joins the cluster as a non-voting member node, does not participate in voting such as cluster elections, and only performs data replication.
After receiving the write request, the leader synchronizes the log to the Follower and Learner nodes, and uses a data structure named Progress in the memory to maintain the log synchronization progress information of the Follower and Learner nodes.
When the difference between the data of the Learner node and the leader's data is small, it can be promoted to a voteable member node to join the cluster.
The advantages of this program are as follows:

After the regional networks are interconnected, the deployment is simple, just add a Learner node to the etcd cluster, no need to operate and maintain additional components
Learner nodes can synchronize any type of data, such as key-value, auth authentication data, lease data
After introducing its advantages, let's take a look at its disadvantages, as follows:

The Learner node only allows serial reading, that is, if the business reads nearby, it will read the old data.
Relying on the higher version of etcd, etcd 3.4 and above only support the Learner feature, and only one Learner node is allowed.
After a complete failure of the main cluster, the Learner node cannot be quickly promoted to a writable independent etcd cluster.
After introducing the several existing solutions, we found that none of them could meet the demands of the business production environment, so we completed the implementation of the etcd synchronization service available in the production environment by ourselves, which will be introduced in detail in the overall solution implementation chapter.

How to ensure the stability and operability of etcd and zk synchronization services?

In order to ensure the stability of etcd and zk synchronization services, 5 types of common failures are simulated, and the self-healing ability of the service under these typical failure scenarios is tested. The detailed test plan is as follows.

#### Failure scenario
Redis flash (depending on the zk2etcd service), for example: redis version upgrade, non-smooth expansion.

zk2etcd is offline, such as OOM, container eviction, host failure.

etcd2etcd is offline, such as OOM, container eviction, host failure

Network interruption, such as OOM, container eviction, host failure.

Weak network environment, such as: temporary replacement with public network after the dedicated line is disconnected.

The actual trigger causes of the above 5 scenarios are varied, and only one situation needs to be simulated.

Exercise plan

Redis flash: By changing the host to simulate redis unreachable, the automatic correction will stop at this time; after the simulated redis is restored, the automatic correction will also automatically resume.

zk2etcd offline: by killing the container node to simulate zk2etcd hanging up, the k8s will be automatically pulled up within 15 seconds, and the synchronization will be normal and the data will be consistent after the pulling up is completed.

etcd2etcd offline: by killing the container node to simulate zk2etcd hanging up, the k8s will be automatically pulled up within 15 seconds, and the synchronization will be normal and the data will be consistent after the pulling up is completed.

Network interruption: Simulate the unreachable zk and etcd by changing the host. At this time, the synchronization is interrupted, and then remove the host to simulate the network recovery. After the restoration, the synchronization is normal and the data is consistent.

Weak network environment: The weak network environment is simulated by switching to the public network. After switching to the public network, the synchronization efficiency is reduced within 4 times, and a full synchronization can still be completed within 1 minute.

In addition, for operability issues, whether it is etcd or zk synchronization services, detailed metrics and logs are provided. We have configured visual observation views for each core scenario and abnormal scenario, and configured alarm policies.

Overall plan landing

Overall structure

The multi-active architecture diagram of etcd cluster is as follows:

illustrate

Black solid line: dedicated line access under normal circumstances

Black dotted line: access through public network

Red solid line: dedicated line access after active/standby switchover of etcd cluster

Red dotted line: public network access after active/standby switchover of etcd cluster

The etcd2etcd/zk2etcd data synchronization service diagram is as follows:

zk synchronization service engineering practice

The storage structure of zookeeper and etcd is inconsistent, which increases the difficulty of synchronization. Zookeeper storage is a tree structure, while etcd v3 is a flat structure. Zookeeper cannot list all keys according to the prefix like etcd; etcd cannot query the child nodes of a certain directory through list chilren like zookeeper, which also increases the difficulty of synchronization.

How to perceive data changes in zookeeper? Zookeeper's watch is not like etcd, which can simply perceive the addition of any key. It needs to recursively watch all the nodes. After receiving the ChildrenChanged event, get all the child nodes under the node corresponding to the event, and then compare it with the data in etcd By comparison, the newly added data can be obtained and put into etcd synchronously. Similarly, you can use the recursive method to watch the deletion events of all nodes and delete the data in etcd synchronously.

In addition, zookeeper's watch has a congenital defect. The watch is a one-off, so it must be watched again after receiving an event. Theoretically, it is possible to lose the event between two watches, mainly in the same key multiple times in a row. It may happen when it changes. If a loss event occurs, data consistency will be destroyed. We have introduced the ability to automatically diff and correct, that is, to calculate the difference between the data in zookeeper and etcd, each time it will go through two rounds of diff calculations, because in the case of frequent data changes , A round of diff calculation often has some "false differences" caused by not strong consistent synchronization. When the diff calculates the result, these differences will be automatically fixed.

How to solve coexistence with etcd2etcd? When there is data written synchronously by etcd2etcd and data written by zk2etcd under the same path, the difference will be calculated and corrected in the automatic correction logic of zk2etcd, but we do not want to delete the data written by etcd2etcd by mistake. data. We have solved this problem by introducing redis for zk2etcd to store state. When zk2etcd writes or deletes data synchronously to etcd, it also synchronously records and deletes in redis:

Then when zk2etcd automatically corrects and calculates the difference, it only considers the data written by this tool to avoid accidentally deleting data written by other synchronization tools.

etcd2etcd engineering practice

In order to solve the problem of etcd synchronization, we investigated the following two solutions, and then we will introduce its principle in detail:

etcd-syncer mirror-plus version

First, we introduce the mirror-plus solution of etcd-syncer. As the name suggests, it is an enhanced version of make-mirror in etcd community. In order to solve the various shortcomings of make-mirror, it achieves the following features and advantages:

Support a variety of synchronization modes, full synchronization, interrupted transmission, no longer worry about the quality of private lines, public network jitter
Highly available, the instance responsible for the same data path replication supports multi-copy deployment. After one copy fails, the other copies will be locked after 5 seconds, and fast recovery is performed based on the progress of the previous instance synchronization
Support consistency check (full data check, snapshot check)
Supports multi-instance concurrent replication to improve performance (different instances are responsible for different paths), it is recommended to configure multiple instances in the production environment, and each instance is responsible for different paths
Good operation and maintenance capabilities, one-click deployment based on k8s deployment, rich metrics, logs, and complete e2e test cases covering core scenarios (http/https scenarios, abnormal service interruptions, network abnormalities, etc.)
So what are its disadvantages? Because its core principle still relies on the mvcc+watch feature of etcd, the data cannot guarantee strong consistency and only synchronize key-value data.

Resumable transfer depends on the retention time of the mvcc historical version, and it is best that the business can save at least 1 hour of historical data.
When the write request is large, there may be a certain amount of data lag in the standby cluster, and dirty data may be read.
It does not support synchronization of non-key-value data, such as auth authentication related data, lease data, etc.

Raft version of etcd-syncer

In order to solve all types of data synchronization problems and eliminate the reliance on etcd mvcc historical data, Tencent Cloud can also provide the Raft version of etcd-syncer, a synchronization solution based on Raft logs.

Its deployment diagram is shown below. The etcd-syncer synchronization service, as a learner node, joins the main etcd cluster.

The leader of the main etcd cluster synchronizes the Raft log data to etcd-syncer through messages such as MsgApp/Snapshot, etcd-syncer parses the Raft log, and applies the Txn/Delete/Auth request corresponding to the Raft log entry to the target etcd cluster.

It has the following advantages:

It has all the features and advantages of the mirror-plus version of etcd-syncer, and does not rely on etcd mvcc historical data.

Based on the Raft log synchronization data at the bottom of etcd, various types of data such as key-value, auth, lease, etc. can be synchronized.

Do not rely on higher versions of etcd.

Complete disaster tolerance test

#### grpc-proxy
This solution introduces the grpc-proxy proxy service and is also used for the first time. In order to understand the performance of this proxy service, we used the benchmark that comes with etcd to perform read and write tests, and we wrote a small tool to do a watch test. The following is part of the test content.

Write test

Directly access the load balancing entrance of the etcd service

Use grpc-proxy to access etcd services

The grpc-proxy proxy can be written normally when the endpoints are configured on a dedicated line or on the public network.
When the total number of write keys is constant, the larger the number of connections and the number of clients, the lower the total time consumption
The larger the total number of write keys, the average time (Average) of a single write will increase, but it is still milliseconds
When the total number of keys written at one time is 100,000, there will be too many requests error when directly connected to etcdserver, but grpc-proxy does not
The performance of the public network is lower than that of the dedicated line
The average time consumption of grpc-proxy proxy is increased compared with direct connection, but it meets the demand
Read test

Directly access the load balancing entrance of the etcd service

Use grpc-proxy to access etcd services

The grpc-proxy proxy can be read normally when the endpoints are configured on a dedicated line or on the public network.

The average time to go to grpc-proxy has increased compared to direct connection, but it is within an acceptable range

watch test

Perform watch test on grpc-proxy according to an etcdwatcher service written by ourselves: you can set the total number of watchers, update frequency, and test time, and print out a briefing report at the end

./etcdwatch -num=100 -span=500 -duration=10 -endpoint=http://grpc-proxy-addr:23791
test done
total 100 task
0 task failed
current revision is 631490
least revision is 631490
0 task is not synced

Parameter Description:

num number of tasks

span update interval, in milliseconds

duration total test time, in seconds

current revision: represents the revision written

least revision: Indicates the revision with the slowest synchronization among num tasks

If failed is 0, it means normal; if task not sync appears, it means watch and put are out of sync

From the above test results: the number of failed is 0, and the watch test is normal

zk2etcd

We are using version 1.2.5, which is deployed through k8s deployment

Simulate zk server disconnection

Scenes
Resolve addresses by injecting errors into hosts

Phenomenon
During the period, no error log of zk lost connection was found
No abnormality found in monitoring indicators
After the restart was performed, the fixed operand did not increase convexly (in version 1.2.4, there is a bug that full sync is executed regularly, but the key that needs to be fixed is not perceived. As a result, after restarting the zk2etcd service instance, it may be observed The phenomenon of fixed operation convex increase)

Simulate redis loss

Simulation operation
09:56:49 Inject redis into the hosts to resolve the address incorrectly
10:07:34 Restore redis
10:16:00 Restart the synchronization service pod (the operation restart is to observe whether full sync is normal)

Phenomenon
During the period, the number of fixed operations did not increase, and no obvious abnormalities were found in other monitoring indicators
There is no convex increase in the fixed number after the instance restarts

Simulate etcd disconnection

Simulation operation
16:15:10 etcd server lost connection

16:30 resume

16:45 restart pod

Phenomenon
During the period, the number of fixed operations did not increase, and no obvious abnormalities were found in other monitoring indicators

After restarting, the number of fixed operations has increased (not sure whether full sync did not take effect or was caused by an update and repair just after the restart)

Summarize

As long as the full sync mechanism works normally, after each abnormal scenario occurs, it can be restored after the next full sync is triggered

The minimum recovery interval depends on the set full sync timing execution interval (the default is 5m), and the business adjusts the parameters for this interval tolerance.

In addition, in order to avoid the occurrence of an abnormality, the full sync mechanism runs regularly, but the situation is not detected, so you can restart the zk2etcd service as soon as possible afterwards.

For the additional etcd public network test, full sync completed and zk, etcd operations take time, which has a certain (second-level) increase compared to the internal network situation

etcd2etcd

For the synchronization service of etcd2etcd, I use deployment dual-copy deployment

Multi-copy backup capability

expect
The backup node will take over the synchronization task after 5s

Test program
etcd syncer dual instance deployment

Kill the running worker node for observation

in conclusion
Whether it is incremental synchronization or full synchronization, the active/standby switchover can work normally (note that when the active/standby switchover occurs during full synchronization, it will become incremental synchronization, which may result in slower comparison)

Resumable transfer capability

expect
After the failure is restored, the synchronization can be continued from the breakpoint

In fact, in Part 1, the standby node took over the synchronization work after switching to the master node. The fast_path changed to 1 also proved the ability of resumable transmission. We also added several additional verification scenarios:

(a) Short-term failure

Failure scenario

During the synchronization process from the central etcd cluster to the hot standby cluster, the key of -etcd-syncer-meta- is also present in the central etcd cluster as the source, which triggers the synchronization service to report an error (the same key cannot be contained in the txn) and appears Data difference

Phenomenon

Add the synchronization service operating parameters to the filtering of -etcd-syncer-meta-, and then observe that after a period of time to catch up with the data, the number of misses finally drops to reach a consistent level

(b) Prolonged failure

Failure scenario

Stop the deployment of the synchronization service

Wait for the data difference between the two etcd clusters and start the synchronization service after a compact

Phenomenon

After the data difference occurs and compaction occurs, restart the synchronization service. The log is as follows: Because compacted occurs, full synchronization is triggered

Synchronization service monitoring indicators: (a) dst miss key dropped quickly; (b) src miss key increased, and continued to not drop

analyze

After the synchronization service is stopped, the number of keys in the source etcd has changed a lot. The monitoring chart shows that there has been a decline during the period, indicating that the key has been deleted.

There is also a small problem exposed here. When the src miss key occurs, it cannot be automatically repaired at present, and manual access is required to clean up the redundant keys.

3. reset triggers full synchronization

When a major synchronization difference (such as a dst miss occurs) for emergency repair, configure the --reset-last-synced-rev parameter to delete the resumable transmission information to trigger the full synchronization to repair the difference

Phenomenon
Due to some kind of abnormality, dst miss (the yellow line example in the figure) appears synchronously. In order to repair, run after adding the --reset-last-synced-rev parameter to the new instance

analyze

slow_path is 1, indicating that full synchronization is triggered (the green line example in the figure)

The dst miss value of the green line instance does not increase, indicating that it has reached the same level

4. Network failure
   Dedicated line interruption between two etcd clusters

Incremental synchronization

Full synchronization

Test program

When the private line is interrupted and switched to the public network, you need to modify the etcd cluster access address in the operating parameters, that is: a restart will occur (the restart scenario test has been covered before, and will not be repeated here)

Summarize

The etcd-syncer synchronization service has a better active and standby mechanism, which can be switched in a timely and effective manner

The performance of the resumable transmission after a short-term failure is in line with expectations; for a long-term failure and the complex situation of compact occurs at the same time, the src miss occurs after the synchronization is restored, and manual access may be required

By configuring the --reset-last-synced-rev parameter, it has a good effect on the abnormal repair of src miss

about us

For more cases and knowledge about cloud native, please follow the public account of the same name [Tencent Cloud Native]~

Welfare: The official account backstage reply [manual], you can get "Tencent Cloud Native Roadmap Manual" & "Tencent Cloud Native Best Practices"~