Comparison of Redis clustering solutions: Codis, Twemproxy, Redis Cluster

As we mentioned earlier, in order to ensure the high availability of Redis, the following aspects are mainly required:

• Data persistence
• Master-slave replication
• Automatic failure recovery
• Clustering

Let's briefly analyze the characteristics of these programs and the relationship between them.

Data persistence is essentially for data backup. With data persistence, when Redis is down, we can restore the data from the disk, but before the data is restored, the service is unavailable, and the time for data restoration Depending on the size of the instance, the larger the amount of data, the slower the recovery. For the persistence process of Redis, please refer to: How does Redis persistence? Comparative analysis of RDB and AOF.

The master-slave replication is to deploy multiple replica nodes, and multiple replica nodes replicate the data of the master node in real time. When the master node is down, we have a complete replica node to use. On the other hand, if the volume of read requests in our business is large and the master node cannot accept all read requests, multiple replica nodes can share read requests and achieve read-write separation, which can improve the access performance of Redis.

But there is a problem. When the master node is down, although we have a complete replica node, we need to manually upgrade the slave node to continue to provide services. If the master node fails, manual operation is required. This process is both Time-consuming and labor-intensive, timeliness cannot be guaranteed, and the degree of high availability will be greatly reduced. How to optimize it?

With data persistence, master-slave replication, and automatic failure recovery when using Redis?

The answer is no. If most of our business is read requests, we can use read-write separation to improve performance. But what if the volume of write requests is also large? Now is the era of big data. Large-scale companies like Ali and Tencent have a very large amount of writes at all times. At this time, if only one master node is unbearable, how to deal with it?

This needs to be clustered ! In simple terms, the implementation is that multiple master and slave nodes form a cluster, and each node stores a part of data, so that write requests can also be distributed to multiple master nodes to solve the problem of high write pressure. At the same time, clustering can dynamically add new nodes when the node capacity is insufficient and the performance is insufficient, and the group can be expanded to improve performance.

From the beginning of this article, we have begun to introduce the Redis clustering solution. Of course, clustering also means that Redis deployment architecture is more complex, and the cost of management and maintenance is also higher. And in the process of use, there will be many problems, which also derives different clustering solutions, and their focus is different.

In this article, we will first introduce several popular solutions for Redis clustering. First, we will have an overall understanding of them. Later, I will specifically analyze the clustering solutions that I am more familiar with.

Clustering solution

To achieve clustering, it is necessary to deploy multiple master nodes. Each master node may also have multiple slave nodes. A cluster composed of such a deployment structure can better bear larger traffic requests and more storage. The data.

Being able to bear more traffic is the most basic function of the cluster. The general clustering solution also includes the functions of data persistence, data replication, and automatic failure recovery mentioned above. These technologies are used to ensure the high performance and high availability of the cluster.

In addition, the excellent clustering solution also realizes the online horizontal expansion function. When the number of nodes is not enough, new nodes can be dynamically added to improve the performance of the entire cluster, and this process is completed online, without business perception.

The mainstream Redis clustering solutions in the industry mainly include the following:

• Client sharding
• Codis
• Twemproxy
• Redis Cluster

They can also with whether the center of the divided , wherein client fragmentation, Redis Cluster belonging to the center-of clustering scheme, CODIS, belonging to a centralized Tweproxy clustering scheme.

Whether it is centralized refers to whether the client accesses multiple Redis nodes directly or operates through a middle-tier Proxy. Direct access is a non-centralized solution, and access through the middle-tier Proxy is a centralized solution , They have their own advantages and disadvantages, which will be introduced separately below.

Client sharding

Client sharding mainly means that we only need to deploy multiple Redis nodes, and how to use these nodes is mainly working on the client.

The client uses a fixed Hash algorithm to calculate the corresponding Hash value for different keys, and then reads and writes to different Redis nodes.

Client Sharding Cluster Mode

Client-side sharding requires business developers to evaluate the business request volume and data volume , and then let the DBA deploy enough nodes to the developers to use.

The advantage of this solution is that it is very convenient to deploy. The number of nodes required for the business can be directly deployed and delivered. The rest requires business developers to write the key request routing logic according to the number of nodes, and formulate a rule, generally fixed The Hash algorithm writes different keys to different nodes, and then reads data according to this rule.

It can be seen that its disadvantage is that the cost of using Redis for needs to write routing rules code to use multiple nodes, and if the pre-assessment of the business data volume is not accurate, the expansion and migration costs of The height is , because after the number of nodes is changed, the node corresponding to the Hash algorithm is no longer the previous node.

Therefore, a consistent hash algorithm was later derived to solve the problem of minimizing data migration and performance when the number of nodes changes.

This kind of client-side fragmentation scheme is generally used for relatively stable business data volume, and will not be used in business scenarios where there will be no substantial growth in the later stage. It is only necessary to evaluate the business data volume in the early stage.

Codis

With the development of business and technology, people increasingly feel that when I need to use Redis, we don’t want to care about the number of nodes behind the cluster. We hope that the Redis we use is a large cluster. When our business volume increases, this large The cluster can to solve insufficient capacity and performance problems .

This method is the server-side sharding solution. The client does not need to care about how many Redis nodes are behind the cluster, but only needs to operate the cluster like a Redis. This solution will greatly reduce the cost of developers. You can only need to pay attention to business logic, and you don't need to care about Redis resource issues.

How can a cluster composed of multiple nodes be used by developers as they would when operating a Redis? This involves how multiple nodes are organized to provide services. Generally, we will add a proxy layer between the client and the server. The client only needs to operate this proxy layer. The proxy layer implements specific request forwarding rules, and then The request is forwarded to multiple nodes behind, so this method is also called a centralized clustering solution, and Codis is a clustering solution implemented in this way.

Proxy cluster mode

Codis architecture diagram

Codis was developed by the former pea pod god of the Chinese people and adopts a centralized cluster solution. Because the proxy layer Proxy is required to forward all requests, the performance requirements of the Proxy are very high. Codis is developed in the Go language, which is compatible with development efficiency and performance.

Codis contains several components:

• codis-proxy: mainly responsible for forwarding the read and write requests
• codis-dashbaord: a unified control center that integrates functions such as data forwarding rules, automatic fault recovery, online data migration, node expansion and contraction, and automated operation and maintenance APIs
• codis-group: Redis Server based on the secondary development of Redis 3.2.8 version, adding asynchronous data migration function
• codis-fe: UI interface for managing multiple clusters

It can be seen that Codis has a lot of components, and its functions are very complete. In addition to the request forwarding function, it also implements online data migration, node expansion and contraction, and automatic fault recovery.

Codis’s Proxy is the component responsible for request forwarding. It maintains specific rules for request forwarding. Codis divides the entire cluster into 1024 slots. When processing read and write requests, it uses the crc32Hash algorithm to calculate the hash value of the key, and then according to The hash value modulates 1024 slots, and finally a specific Redis node is found.

The biggest feature of Codis is that it can be expanded online, without affecting client access during the expansion period, that is, no downtime is required. This is a great convenience for business users. When the performance of the cluster is insufficient, nodes can be dynamically added to improve the performance of the cluster.

In order to achieve online expansion and ensure reliable performance of data during the migration process, Codis has modified Redis and added commands for asynchronous data migration. It is developed based on Redis 3.2.8, and the upper layer cooperates with Dashboard and Proxy components to complete Data migration and capacity expansion without any damage to the business.

Therefore, if you want to use Codis, you must use its built-in Redis, which means that whether Redis in Codis can keep up with the latest version of the official features may not be guaranteed. This depends on the maintainer of Codis. At present, Codis has It is no longer maintained, so Redis version 3.2.8 can only be used when using Codis, which is a pain point.

In addition, because clustering requires the deployment of multiple nodes, operating a cluster cannot completely implement all functions like operating a single Redis. The main reason is to disable or restrict commands that may cause problems when operating multiple nodes. For details, please refer to Codis List of supported commands.

But this does not affect that it is an excellent clustering solution. Since our company used the Redis cluster solution earlier, Redis Cluster was not mature enough at that time, so the Redis cluster solution used by our company is Codis.

At present, my work mainly revolves around Codis. Our company has customized development of Codis and also modified Redis so that Codis supports data synchronization across multiple data centers. Therefore, I am more familiar with the code of Codis. Later, I will write some articles to analyze the implementation principle of Codis and learn its principles. This is of great help to our understanding of distributed storage!

Twemproxy

Twemproxy is a clustering solution open sourced by Twitter. It can be used as both Redis Proxy and Memcached Proxy.

Its function is relatively single, it only realizes request routing and forwarding, and does not have the online expansion function as comprehensive as Codis. The focus of its solution is to unify the logic of client fragmentation to the Proxy layer, and other functions do not do any processing.

Tweproxy has been launched for the longest time. In the early days when there was no good server-side fragmented cluster solution, it had a wide range of applications and extremely stable performance.

But its pain point is the inability to expand and shrink online, which leads to very inconvenient operation and maintenance, and there is no friendly operation and maintenance UI to use. Codis was derived from this background.

Redis Cluster

When adopting the centralized mode of adding a layer of Proxy in the middle, this places high demands on the Proxy, because once it fails, all the clients operating the Proxy cannot handle it. To achieve high availability of the Proxy, additional requirements are needed. The mechanism to achieve, such as Keepalive.

Moreover, adding a layer of Proxy for forwarding will inevitably result in a certain performance loss, so besides client sharding and the centralized solution mentioned above, is there a better solution?

The Redis Cluster officially launched by Redis takes a different approach. It does not use a centralized proxy scheme, but puts part of the request forwarding logic on the client side and part on the server side, and they cooperate with each other to complete the request processing.

Redis Cluster was launched in Redis 3.0. The early Redis Cluster has not been widely promoted because it has not undergone rigorous testing and production verification. It is also in this context that the industry has derived the centralized cluster solution mentioned above: Codis and Tweproxy.

However, with the iteration of Redis, the official Redis Cluster has become more and more stable, and more people have begun to adopt the official clustering solution. It is also because it is officially launched, so its continuous maintainability can be guaranteed, which is more advantageous than those third-party open source solutions.

Redis Cluster does not have an intermediate Proxy proxy layer, so how to forward requests?

Redis puts the request forwarding logic in the Smart Client. If you want to use Redis Cluster, you must upgrade the Client SDK. This SDK has built-in request forwarding logic, so business developers do not need to write forwarding rules by themselves. Redis Cluster uses 16384 Each slot is used to forward routing rules.

Without the Proxy layer for forwarding, the client can directly operate the corresponding Redis node, thus reducing the performance loss of Proxy layer forwarding.

Redis Cluster also provides online data migration, node expansion and shrinkage and other functions . It also has a built-in sentinel to complete the automatic recovery function. It can be seen that it is a Cluster that integrates all functions. Therefore, it is very simple to deploy, does not need to deploy too many components, and is extremely friendly to operation and maintenance.

Redis Cluster also handles the client's request processing during node data migration, expansion and shrinking. When the data accessed by the client happens to be in the process of migration, the server and the client have developed some agreements to inform the client to access the correct node and help the client correct its own routing rules.

Although Redis Cluster provides the function of online data migration, its migration performance is not high. When a large key is encountered during the migration process, it may block the two migrated nodes for a long time. This function is compared with Codis. Data migration performance is better. It's probably good to know one here first, and later I will write some articles specifically for the performance comparison of Codis and Redis Cluster online migration functions.

Now more and more companies are beginning to adopt Redis Cluster. Competent companies have also carried out secondary development and customization on the basis of it to solve some problems of Redis Cluster. We look forward to the better development of Redis Cluster in the future.

to sum up

After comparing these clustering solutions, let's summarize them below.

The mainstream clustering solutions in the industry are the above, and a brief introduction to their characteristics and differences. We can choose our own suitable clustering solutions during the development process, but it is best to understand their implementation principles and use them. You can solve problems more calmly when you encounter problems.

Source: kaito-kidd.com/2020/07/07/redis-cluster-codis-twemproxy