The road to improvement of cloud Redis

As a professional cloud service provider that launched the first programmable CDN service in China, Paiyun takes advantage of the scale and performance of the CDN edge network to allow customers to customize and write rules to meet common business scenarios. In order to ensure that these source data, such as edge redirection, request speed limit, custom error page, access anti-leech control, HTTP header management, etc., can be quickly synchronized to the edge node server, after comparing multiple solutions, and Paiyun started to use Redis 2.8 as a data synchronization solution in early 2014.

The initial architecture is as follows:

Before continuing to talk about Redis improvements, we need to understand technical debt. The technical debt mentioned here refers to technical debt. Usually, in order to accelerate software development, developers may compromise when they should adopt the best solution, and switch to a solution that can accelerate software development in the short term. And this solution brings additional development burden to itself in the future. This option, which looks like a benefit in the present, must be repaid in the future, is like debt, so it is called technical debt.

And the solution we mentioned above buried the introduction of technical debt. Although it has played an important role in the past few years, the shortcomings of the architecture are obvious, and with the increase in the number of edge servers and the amount of synchronized data, coupled with the aging failure of server hardware and other reasons, it has caused many problems. For example the following questions:

• For security reasons, the communication data between Redis needs to be encrypted, but Redis itself does not support SSL encryption. Therefore, all edge servers must be used as transit servers through stunnel sockets. However, in the actual working state, the performance of stunnel is insufficient, resulting in high server CPU load.
• The data masters and slaves of Redis have long connections and try to maintain synchronization from the same source. Therefore, early edge servers obtained the IP address of the source server through domain name resolution. The advantage of this is that the implementation and deployment are simple, but the disadvantage is that the DNS cannot know the processing capacity of the backend server, resulting in an unbalanced load of long connections on each machine. Moreover, DNS cannot be processed automatically after the back-end service fails. Even if the DNS is switched and resolved in time, the data will be different due to the vacuum period before the TTL takes effect, so that only the old data can be used for emergencies.
• Due to historical reasons, the edge Redis versions are mostly 2.x lower versions, and the lower versions can only do full synchronization through sync. Therefore, the abnormality of the transit server and the main server will cause an avalanche effect of the entire network, which will block synchronization and fail to quickly synchronize metadata to the edge.
• Because in the early days, only the master-slave mode could be used in Redis, and sentinel and cluster transformation were not implemented. So making today's primary server a single point of risk, it's easy to cause major failures at the source.

Because of the problems and side effects caused by the previous compromises, we now have to spend extra time and effort to refactor and improve the architecture to the best way to achieve it.

We divide the transformation process into several steps:

Strengthen the security protection of SSL and upgrade to the latest stable version of OpenSSL as much as possible

SSL may be one of the Internet security protocols that everyone is exposed to more. Generally, the website address starts with "https://", which means that the SSL security protocol is used. OpenSSL is an open source SSL implementation used to implement high-strength encryption of network communications, and is now widely used in various network applications. Such an important project has been underfunded and understaffed for many years, and most of the work has to be done by a small number of hackers and hobbyists and volunteers. Fortunately, it is now included in the funding target of the Linux Foundation, but there are still new vulnerabilities that are constantly exposed, which require timely attention and follow-up.

Refer to the latest OpenSSL vulnerability risk rating report:

Given that there are too many security holes in the RC4 algorithm, it is recommended to disable it when compiling.

Use the latest stunnel version, optimize performance, based on secure OpenSSL dependency library, support TLSv1.2+ and above

As can be seen from the red box in the figure below, stunnel has the strongest performance under certain algorithms, so it is recommended to use it first in the configuration file:

 ./configure --prefix=/opt/stunnel --with-ssl=/opt/openssl

Take a look at the optimization options in the recommended configuration:

 verify = 3
sslVersionMax = TLSv1.3
sslVersionMin = TLSv1.2
options = NO_SSLv2
options = NO_SSLv3
.......
ciphers = ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4:!DH:!DHE

The trust level detection and verification of HTTPS can be done through the website of Asia Integrity.

Compile the latest stable version of Redis-6.2.x, which is powerful and rich in functions and does not need to depend on a higher version of GCC

Comparing Redis 6.2 with 7.0, the 7.0 version is definitely more powerful. Redis 7.0 includes incremental improvements to almost every aspect, most notably Redis Functions, ACLv2, command introspection, and Sharded Pub/Sub. Version 7.0 adds nearly 50 new commands and options to support this evolution and extend Redis' existing functionality.

But although Redis7.0 is more powerful, we finally chose Redis6.2 considering the full compatibility with the original Redis code and the stability of the production environment. Because Redis6.2 has enough advantages and powerful functions, and can better meet the requirements of our production environment, such as:

• Multithreaded IO (Threaded I/O)
• Numerous new modules API
• Better expiration cycle
• SSL support
• ACLs access control
• RESP3 protocol
• Client side caching
• Diskless Copy & PSYNC2
• Redis-benchmark supports clustering
• Redis-cli optimizes, rewrites Systemd support
• Redis Cluster Agent released with Redis6 (but in a different repo)
• RDB loads faster
• SRANDMEMBER and similar commands have better distribution
• STRALGO command
• Redis commands with timeouts are easier to use

Focus on the features of PSYNC2, which is also one of the key features of our architecture improvement and upgrade.

In the actual production operation of Redis cluster, the maintenance restart of the instance and the failover of the master instance (such as cluster failover) are relatively common (such as instance upgrade, rename command, and release of instance memory fragments, etc.). Before the Redis 4.0 version, such maintenance processing Redis will be fully re-synchronized, resulting in a small amount of performance-sensitive services being damaged. PSYNC2 mainly allows Redis to use partial resynchronization in the scenario of slave instance restart and master instance failover.

Download the source code directly to compile:

 # make BUILD_TLS=no

Recommended configuration, add the following options to enhance performance:

 io-threads-do-reads yes
io-threads 8
aof-use-rdb-preamble yes

During our testing, we found several problems with Redis+TLS:

• After Redis turned on TLS, the performance dropped by 30%.
• Redis has a strong dependence on OpenSSL. Considering the high-risk vulnerabilities of OpenSSL in the past, if you want to constantly fix the vulnerabilities, you need to recompile Redis, which will lead to high cost of operation and maintenance updates.
• After Redis is upgraded, it is necessary to resynchronize data, which increases the probability of failure or stops production.

Therefore, we still decided to use the third-party program stunnel to strengthen security, facilitate upgrades and fix vulnerabilities. It does not affect the back-end connection, thus ensuring the continuity and stability of Redis's work.

Based on APISIX+TLS hosting, using TCP hash consistency for load balancing to replace DNS polling, the performance is remarkable

APISIX uses TCP proxy. This part can be used after direct configuration. It has little to do with Redis transformation. We will skip it directly. You can directly look at the screenshot of connection statistics after transformation. From the actual number of APISIX connections, it can be seen that the load is evenly distributed to different backends, and the restart of the edge server also uses PSYNC2 to perform fast incremental synchronization.

Customized data synchronization using Redis-shake

In the process of architecture improvement, we also looked at the tool redis-shake, which is an open source tool for Redis data synchronization by the Alibaba Cloud Redis&MongoDB team. It supports four functions: analysis, recovery, backup, and synchronization . To give you the main introduction to synchronization sync:

Restore restore: restore the RDB file to the destination Redis database.

Backup dump: Back up the full data of the source Redis through the RDB file.

Parse decode: Read the RDB file and parse and store it in json format.

Synchronization sync : Supports data synchronization between source Redis and destination Redis, and supports full and incremental data migration.

Synchronous rump: Supports data synchronization between source Redis and destination Redis, and only supports full migration. The scan and restore commands are used for migration, and the migration of different Redis versions of different cloud vendors is supported.

We used to have a Redis with source code modifications that would only synchronize the desired space. Although it is easy to use, it still needs to be recompiled on the new code, but the original person in charge can no longer be found. This is also a common problem for many years of disrepair projects, but through open source tools such as redis-shake, we can achieve the functions we want with a simple configuration through it:

 - filter.db.whitelist / blacklist
  - filter.key.whitelist / blacklist
  - filter.command.whitelist / blacklist

Current Architecture and Future Prospects

In the current architecture, we have split and strengthened the three-tier architecture based on the original three-tier architecture:

• The DNS layer resolves to VIP, which uses the dynamic gateway routing protocol of BGP/OSPF and corresponds to the latter group of server cluster services.
• Load balancing layer: Use "apisix" + "tls1.2+" + "tcp hash consistent connection" to balance the master-slave connection of Redis and failover.
• Edge CDN nodes use the technical dividends brought by the high version of Redis, the incremental synchronization of psync, and stunnel+tls1.2 to realize encrypted transmission.

In the next stage, we will continue to transform the Redis master of the data center into the Redis sentinel mode (considering that the program code needs to be modified for compatibility with the sentinel mode, the first stage will not come first, everything is for the stability in the production environment).

Reference documentation:

How to check the TLS version of a website: https://wentao.org/post/2020-11-29-ssl-version-check/

Redis Features Replication Enhanced PSYNC2: https://www.modb.pro/db/79478

A detailed explanation of the easy-to-understand Redis architecture pattern: https://www.cnblogs.com/mrhelloworld/p/redis-architecture.html

Recommended reading

[Practical and dry goods] Do these 16 optimizations well, and your Linux operating system will look brand new

Singleton pattern of common design patterns in Golang