Long-form illustration etcd core application scenarios and coding practice

Hello everyone, I am Antetokounmpo. I am writing an article about etcd today. In fact, there are many introductions about etcd on the Internet. I will be brief and concise. I hope you can master the core knowledge and coding skills of etcd through this article. !

• This article first uses the vernacular to introduce to you what etcd is? This part of the content has been a lot of online.
• What are the application scenarios of etcd? What are the core principles of these application scenarios?
• In the end, you can't just move your mouth without moving your hands. First build an etcd stand-alone version, and then use the java client to operate etcd data.

This article aims to help you understand etcd, view etcd from a macro perspective, and master the basic operation skills of etcd. In the follow-up, I will write a series of articles to code each application scenario. I look forward to everyone paying attention to me and my official account: Antetokounmpo. Subsequent plan chapters are as follows:

• "Building a highly available etcd cluster"
• "Implementing distributed locks based on etcd (java code implementation)"
• "Implementation of configuration change notification based on etcd (java code implementation)"
• "Realizing service registration and discovery based on etcd (java code implementation)"
• "Implementation of distributed system node leader election based on etcd (java code implementation)"

1. Vernacular etcd and zookeeper

Friends who have used linux, please raise your hand, okay, I saw it! All automatically installed system software configuration files in linux are stored in a directory named /etc . "d" means distributed distributed, etcd is a distributed model, so the core application scenario of etcd is: configuration information storage of distributed systems .

The first sentence of many articles on the Internet is copied from the English official website: etcd is a highly consistent distributed key-value storage system . Many friends asked after reading it, what is the difference between this thing and redis? The author wants to say, really don’t compare it like this, etcd has already told you from the name, it stores configuration information (metadata). It is not at the same level as redis in terms of architecture application. Its target product should be zookeeper. Although zookeeper is widely used in many Java distributed systems, etcd, as a rising star, has a tendency to surpass zookeeper by taking advantage of kubernetes.

• zookeeper is written in java, etcd is written in go language. Zookeeper uses the TCP protocol, and its interactive message rules are completely customized. If you do not use the SDK provided by zookeeper, you cannot operate data. And etcd uses google's gRPC protocol, which is more universal.
• Zookeeper opens a socket to listen for a request. And etcd's monitoring pipeline channel can be used repeatedly. From the perspective of IO performance and system resource utilization, etcd is undoubtedly better.
• Zookeeper uses the zab protocol to ensure the consistency of cluster node configuration information, and etcd uses the raft protocol. If you want to learn more about the raft protocol, click "raft protocol introduction in Chinese" .

Most of the functions are the same as zookeeper. Currently, java programmers use zookeeper more, and other programmers use etcd more. It's all based on habit, but I recommend etcd.

2. The 4 core mechanisms of etcd

etcd stores data in the form of key-value. With the following four mechanisms, the application scenarios of etcd are more extensive.

• Prefix mechanism : the prefix mechanism, also known as the directory mechanism , the client puts 2 key-value pair configurations into etcd, if one key is "/test/key1" and the other key is "/test/key2". Then pass the prefix "/test" queries etcd and returns a list of key-value pairs with keys "/test/key1" and "/test/key2";
• Watch mechanism : that is, the monitoring mechanism. The watch mechanism monitors a certain key and also supports range monitoring for prefixes. When the monitored key or prefix range changes, the client will receive a change notification;
• Lease mechanism : the lease mechanism (TTL, Time To Live), supports adding a time-to-live for the key-value, beyond which the key-value will expire and be deleted. It supports contract cancellation (deleting key-value), renewal (increasing TTL time) ) and so on.
• Revision mechanism : Each key has a globally unique Revision number, and each transaction is incremented by 1, which is globally unique, so the order of data write operations can be determined through Revision, which is very helpful for implementing distributed locks and queues.

3. Leader election and client interaction

When using etcd, in order to ensure high availability, cluster deployment is usually used. Deploy an odd number of nodes, usually 3 or 5 nodes are recommended, because the etcd clusters need to interact through the network to ensure the consistency of configuration information . Distributed multi-nodes ensure high availability, but too many nodes are not good. The more nodes, the greater the network consumption. As for why the odd number? This involves the issue of leader election, and the odd number is convenient to vote for the result.

etcd uses the raft algorithm to ensure data consistency between nodes in the cluster. The raft algorithm divides the nodes in the cluster into three roles: Leader, Follower, and Candidate.

• When the cluster is initialized, each node is a Follower. A node is elected as the leader by voting through the raft algorithm.
• As the master node, the Leader maintains a heartbeat with other nodes and synchronizes data to other nodes.
• When a follower does not receive the leader's heartbeat for a period of time, it will change its role to a candidate candidate and initiate a new election to elect a new leader.

When the client operates etcd cluster data:

• Read operation: The client can access any node for data read operation
• Write operation: The client accesses any node for write operation, and if the node is a Follower, it forwards the request to the Leader. The leader is responsible for the data write operation (addition, deletion and modification), persists the data, and sends a message to synchronize the data to the follower.

Fourth, the application scenarios of etcd

4.1. The kubernetes brain

At present, the most typical application scenario of etcd is as the brain of a Kubernetes cluster.

If kubernetes is compared to a big hotel, etcd is the invoicing + customer relationship management system of the hotel.

• As a container orchestration service, kubernetes performs reasonable resource allocation and service orchestration for various services provided to customers.
• Inevitably, there is some kubernetes cluster configuration and state data, such as the number of pods, their state, namespace, etc. There needs to be a unified record and management place, which is etcd.

The most important thing is: etcd has the function of watch monitoring. Once a configuration or a state changes, all services in the cluster can obtain messages in real time through the watch monitoring mechanism, and then make further responses. Almost all application scenarios of etcd are generated based on the watch monitoring mechanism, including service registration discovery and subscription notifications that we will introduce to you later.

4.2. Service Registration and Discovery

In fact, kubernetes also uses etcd to realize the service registration discovery mechanism, but the above picture is not very easy to explain. I have drawn two new pictures to illustrate the role of etcd in realizing the service registration discovery mechanism.

The so-called service registration implementation principle is: when the service is started, a piece of configuration data is written to etcd, and the configuration data indicates its own service name, service ip address, service port and other information.

Example of the so-called service discovery implementation principle: an instance of service C wants to access service A, service C asks etcd for the access address of service A, etcd responds with the result: there are three instances of service A, and the address list is as follows: xxx.xxx.xxx.xxx:端口 , yyy.yyy.yyy.yyy:端口 , zzz.zzz.zzz.zzz:端口 . Service C does not need to access three instances, it can get the result by accessing one of them, so it selects one according to its own load balancing algorithm, which is called: client load balancing.

4.3. Health Check and Status Change Notification

Link to the above: When service C accesses service A next time, does it need to access etcd? The answer is no , after it has accessed it once, it will maintain a list of service A access addresses by itself. Unless the list changes, it will not ask etcd again.
So how does one service know that another service's list has changed? For example, the instance registration status of service A changes. It may be hung up for some reason, it may be OOM or network problems, etc.

• After a service is registered with etcd, it will save a registration configuration information about the service. The registration configuration information is set by a TTL, and etcd will maintain a heartbeat with the service at the same time. Once the TTL time is exceeded and the heartbeat response from the service cannot be obtained, etcd considers that there is a problem with the health status of the node, and will take the node offline (delete the registration and configuration information).
• After the service is registered with etcd, it will keep monitoring the status data changes of etcd. Once the monitoring result is obtained: the instance status of service A changes, the service will re-pull the registration list of service A from etcd.

4.4. Distributed lock

Cross-process and cross-system multi-threaded operation of common resources, multi-threaded competition occurs, in order to avoid thread insecurity, it is necessary to use distributed locks. If multiple threads compete for resources within a single process, just use Lock instead of distributed locks. For example, if you have a user balance data in the mysql library, and threads in multiple processes change this value at the same time, concurrent data coverage may occur. In order to avoid such a problem, multiple processes are queued up, A comes first, A releases the lock and B comes again, and B releases the lock and C comes again.

Example: The three clients in the above figure represent three services, all of which must operate a certain resource data.

• When trying to call the locking API, the revision=1 obtained by client1 makes it eligible for locking first. Locking is to add a configuration record with revision. All other services monitor the release of locks through the watch mechanism.
• The client was assigned revision when trying to call the locking API. And sorted according to revision, the monitoring distance is the smallest difference between your revision, and it is smaller than your own revision, and there will be no shocking herd effect.

4.5. Implement message queue (pure nonsense)

I think using etcd to implement message queues is a pure nonsense approach. If you have any objections, please leave a message!

It's not that it can't be done, it is indeed possible to write a demo. Putting data into etcd and monitoring it through the watch mechanism is not a typical message queue? Bullshit! If I only want to realize the publish and subscribe of message data, there are actually many ways. I still need to build an etcd cluster? Spring's Event mechanism, Java's reactive programming, even if you build a BlockQueue yourself, can you realize the publish and subscribe of messages?

The reason why we use message queues such as kafka and RocketMQ must be because our asynchronous data has reached a certain scale. Asynchronous message data delivery at scale is not the application scenario of etcd at all, as mentioned at the beginning of this article: don't forget that it is called etcd, it is a storage configuration information for distributed systems, not message middleware.

Five, etcd installation

This article installs a stand-alone version of etcd that can be used in the experimental environment. We can use it for experiments. I will also write an article to introduce the installation method of etcd cluster in the future.
Download the etcd installation package and visit github-etcd . I am using a 64-bit linux operating system, so the downloaded installation package is: etcd-v3.5.4-linux-amd64.tar.gz . If the network conditions do not allow, you can search "etcd domestic download acceleration", select the appropriate download and installation package to install.

First decompress the installation package, after decompression, cd into the installation directory, copy the etcd and etcdctl commands to the /usr/local/bin/ directory.

 tar zxvf etcd-v3.5.4-linux-amd64.tar.gz;
cd etcd-v3.5.4-linux-amd64;
cp etcd etcdctl /usr/local/bin/;

Use the etcd --version command to check the etcd version and verify the installation result. If you don't want to type the full path, you can add the /usr/local/bin directory to the system's PATH environment variable.

 /usr/local/bin/etcd --version

Start etcd, the role of listen-client-urls and advertise-client-urls configuration here is to allow remote connections, 0.0.0.0 means to listen to all ips of the current server, and the listening port is 2379. If your server has multiple Block network card, multiple fixed ips, if you want to specify etcd service to provide services on a certain ip, you can use this ip to replace 0.0.0.0

 /usr/local/bin/etcd  --listen-client-urls 'http://0.0.0.0:2379' --advertise-client-urls 'http://0.0.0.0:2379'

After etcd is started, you can add configuration to etcd through the etcdctl command. Use the put command to add a key-value pair data of key=/dir1 , value=aaa as shown below. You can use the get command to get the configuration information.

 # /usr/local/bin/etcdctl put /dir1 aaa
OK
# /usr/local/bin/etcdctl get /dir1
/dir1
aaa

Six, jetcd coding to achieve configuration management

The following is an introduction to the operation of etcd data through the java API. First, jetcd is introduced through the coordinates of maven. The version I use is relatively old. The latest version is already 0.7.8, but when I use it, it appears with the netty version. Inconsistency, an error is reported: Some classes related to netty cannot be found. So I went back to version 0.3.0, and the usage is the same.

 <dependency>
    <groupId>io.etcd</groupId>
    <artifactId>jetcd-core</artifactId>
    <version>0.3.0</version>
</dependency>

The following code uses jetcd to operate etcd's configuration data, and implements data write, read, and delete operations. Please see the code for detailed usage. The following code is the writing method of the unit test case of Junit 5.

 import io.etcd.jetcd.ByteSequence;
import io.etcd.jetcd.Client;
import io.etcd.jetcd.KV;
import io.etcd.jetcd.kv.GetResponse;
import io.etcd.jetcd.kv.PutResponse;
import org.junit.jupiter.api.*;

import java.nio.charset.StandardCharsets;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;

import static junit.framework.TestCase.assertNotNull;

//这个注解配合函数的Order注解,决定测试用例函数的执行顺序
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class EtcdTest {
  private static Client etcdClient;

  @BeforeAll
  static void  init(){
    etcdClient = Client.builder()
             //这里的etcd服务列表可以写多个,用逗号分隔
            .endpoints("http://192.168.161.3:2379".split(","))
            .build();
  }

  @Test
  @Order(1)
  @DisplayName("etcd写配置操作")
  void putKV() throws ExecutionException, InterruptedException {
    KV kv = etcdClient.getKVClient();
    ByteSequence key = ByteSequence.from("key-str", StandardCharsets.UTF_8);
    ByteSequence value = ByteSequence.from("value-str", StandardCharsets.UTF_8);
    //put key-value配置信息
    CompletableFuture<PutResponse> putRsp = kv.put(key,value);
    assertNotNull(putRsp.get().getHeader());
  }


  @Test
  @Order(2)
  @DisplayName("etcd读配置操作")
  void getKV() throws ExecutionException, InterruptedException {
    KV kv = etcdClient.getKVClient();
    ByteSequence key = ByteSequence.from("key-str", StandardCharsets.UTF_8);
    //通过key获取值
    CompletableFuture<GetResponse> getRsp = kv.get(key);
    String getBackValue = getRsp.get().getKvs().get(0).getValue().toString(StandardCharsets.UTF_8);
    System.out.println("从etcd通过key获取value值为：" + getBackValue);
  }


  @Test
  @Order(3)
  @DisplayName("删除配置操作")
  void deleteKV() {
    KV kv = etcdClient.getKVClient();
    ByteSequence key = ByteSequence.from("key-str", StandardCharsets.UTF_8);
    //通过key删除数据
    kv.delete(key);
  }
}

The above code only introduces the most basic key-value operations of etcd. In fact, the etcd client also provides many APIs, which will be covered in my follow-up articles: distributed locks, service registration discovery, configuration change monitoring, distributed It will be introduced to you in the content of system leader election.

 //租约
Lease lease=etcdClient.getLeaseClient();
//监听
Watch watch =etcdClient.getWatchClient();
//选举
Election election =etcdClient.getElectionClient();
//锁
Lock lock=etcdClient.getLockClient();

Welcome to pay attention to my announcement number: Antetokounmpo, reply 003 and present the PDF version of the author's column "The Way of Docker Cultivation", more than 30 high-quality docker articles. Antetokounmpo Blog: zimug.com