Recommend collection! See the complete overview, the most detailed Redis summary (the latest version of 2021)

Introduction to Redis

Redis is an open source (BSD license) high-performance non-relational (NoSQL) key-value database written in C language.

Redis can store mappings between keys and five different types of values. The key type can only be a string, and the value supports five data types: string, list, set, hash table, and ordered set.

Unlike traditional databases, Redis data is stored in memory, so the read and write speed is very fast. Therefore, redis is widely used in the cache direction. It can handle more than 100,000 read and write operations per second, which is the fastest known performance Key-Value DB. In addition, Redis is often used for distributed locks. In addition, Redis supports transactions, persistence, LUA scripts, LRU-driven events, and multiple cluster solutions ( Redis 6.0 cluster construction practice ).

From March 15, 2010, the development of Redis is hosted by VMware. Since May 2013, the development of Redis has been sponsored by Pivotal.

Pros and cons of Redis

advantage

• The read and write performance is excellent. Redis can read at a speed of 110,000 times/s and write at a speed of 81,000 times/s.
• Support data persistence, support AOF and RDB two persistence methods.
• Support transactions, all operations of Redis are atomic, and Redis also supports the atomic execution of several operations after merging.
• Rich data structure, in addition to supporting string type value, it also supports hash, set, zset, list and other data structures.
• Supports master-slave replication, the master will automatically synchronize data to the slave, and read and write can be separated.

Disadvantage

• The database capacity is limited by physical memory and cannot be used for high-performance reading and writing of massive data. Therefore, the suitable scenarios for Redis are mainly limited to high-performance operations and calculations with a small amount of data.
• Redis does not have automatic fault tolerance and recovery functions. The downtime of the host and slave will cause some of the front-end read and write requests to fail. You need to wait for the machine to restart or manually switch the front-end IP to recover.
• The host is down, and some data cannot be synchronized to the slave in time before the downtime. After switching the IP, data inconsistency will be introduced, which reduces the availability of the system.
• Redis is more difficult to support online expansion, and online expansion will become very complicated when the cluster capacity reaches the upper limit. In order to avoid this problem, the operation and maintenance personnel must ensure that there is enough space when the system is online, which causes a great waste of resources.

type of data

Redis mainly has 5 data types, including String, List, Set, Zset, and Hash, which meet most of the usage requirements. For details, please refer to: Redis's 8 data types, very well written!

scenes to be used

Due to Redis's excellent read and write performance, persistence support and other advantages, Redis has many usage scenarios, including counters, caches, message queues, distributed locks, etc. The specific usage scenarios are as follows:

• counter

  • You can perform self-increment and self-decrement operations on String to realize the counter function.
  • The read and write performance of an in-memory database like Redis is very high, and it is suitable for storing frequent read and write counts.

• Cache

  • Put the hot data in the memory, set the maximum memory usage and elimination strategy to ensure the cache hit rate.

• Session cache

  • You can use Redis to uniformly store the session information of multiple application servers.
  • When the application server no longer stores the user's session information, it no longer has state. A user can request any application server, which makes it easier to achieve high availability and scalability.

• Full page cache (FPC)

  • In addition to the basic session token, Redis also provides a very simple FPC platform.
  • Take Magento as an example. Magento provides a plug-in to use Redis as a full-page cache backend. In addition, for WordPress users, Pantheon has a very good plugin wp-redis, which can help you load the pages you have browsed as quickly as possible.

• Lookup table

  • For example, DNS records are very suitable for storage using Redis.
  • The lookup table is similar to the cache, but also uses the fast lookup feature of Redis. However, the content of the lookup table cannot be invalidated, and the cached content can be invalidated because the cache is not a reliable source of data.

• Message queue (publish/subscribe function)

  • List is a doubly linked list, you can write and read messages through lpush and rpop
  • However, it is better to use messaging middleware such as Kafka and RabbitMQ.

• Distributed lock implementation

  • In a distributed scenario, locks in a stand-alone environment cannot be used to synchronize processes on multiple nodes.
  • You can use the SETNX command that comes with Redis to implement distributed locks. In addition, you can also use the official RedLock distributed locks.

• other

  • Set can realize operations such as intersection, union, etc., so as to realize functions such as mutual friends.
  • ZSet can achieve orderly operations, thereby achieving functions such as rankings.

Persistence

Redis is an in-memory database. In order to reuse data later (such as restarting the machine or recovering data after a machine failure), or to back up data to a remote location to prevent system failures, the data in the memory needs to be persisted to the hard disk.

Redis provides two persistence methods, RDB and AOF. By default, only RDB is enabled. When Redis restarts, it will give priority to using AOF files to restore data sets.

For a detailed explanation of Redis persistence, please refer to: Redis persistence

Deletion strategy for expired keys

There is a function of setting time expiration in Redis, that is, an expiration time can be set for the value stored in the redis database. As a cache database, this is very practical. For example, the token or some login information in our general projects, especially the SMS verification code, are time-limited. According to the traditional database processing method, it is generally judged to expire by oneself, which will undoubtedly seriously affect the performance of the project.

Redis has three different deletion strategies: immediate deletion, lazy deletion, and scheduled deletion

• (1): Delete immediately. When setting the expiration time of the key, a callback event is created. When the expiration time is reached, the time processor automatically executes the key deletion operation.
• (2): Lazy deletion. The key expires, no matter what. Every time you get a value by key from the dict dictionary, first check whether the key has expired. If it expires, delete it and return nil. If it does not expire, return the key value.
• (3): Delete regularly. Periodically, check the expires dictionary and delete the expired keys in it.

It can be seen that the second type is passive deletion, the first and third types are active deletion, and the first type is more real-time. The following is a detailed analysis of these three deletion strategies.

• Delete now

Immediate deletion can ensure the maximum freshness of the data in the memory, because it guarantees that the expired key value will be deleted immediately after the expiration, and the memory occupied by it will be released accordingly. But deleting immediately is the most unfriendly to the cpu. Because the deletion operation will take up the CPU time, if it happens to happen when the CPU is very busy, such as when doing calculations such as intersection or sorting, it will cause additional pressure on the CPU.

In addition, the current redis event processor's processing method for time events is an unordered linked list. The time complexity of looking up a key is O(n), so it is not suitable for processing a large number of time events.

• Lazy deletion

Lazy deletion means that after a key value expires, the key value will not be deleted immediately, but will not be checked until it expires until the next time it is used, and then it can be deleted. So the disadvantage of lazy deletion is obvious: waste of memory. Both the dict dictionary and the expires dictionary must store the key value information.

For example, for some data that is updated by time, such as log logs, it may not be accessed for a long period of time after expiration. In this way, bye bye and waste so much memory to store logs. . This is fatal for redis whose performance is very dependent on the memory size.

• Timed delete

From the above analysis, immediate deletion will take up a lot of cpu in a short time, and lazy deletion will waste memory for a period of time, so regular deletion is a compromise.

Timed deletion is to perform a deletion operation at regular intervals, and reduce the impact of the deletion operation on the cpu by limiting the duration and frequency of the execution of the deletion operation. On the other hand, regular deletion also effectively reduces the memory waste caused by lazy deletion.

• Strategies used by Redis

The expiration key value deletion strategy used by redis is: lazy deletion plus periodic deletion, and the two are used in conjunction.

Data elimination strategy

The maximum memory usage can be set, and when the memory usage exceeds, the data elimination strategy will be implemented.

Redis specifically has 6 elimination strategies:

As an in-memory database, for performance and memory consumption considerations, Redis's elimination algorithm is not actually implemented for all keys, but a small part of it is sampled and the eliminated keys are selected.

Redis 4.0 introduces volatile-lfu and allkeys-lfu elimination strategies. The LFU strategy eliminates the least frequently accessed key-value pairs by counting access frequency.

You need to choose an appropriate elimination strategy based on the characteristics of the system. Of course, you can also dynamically set the elimination strategy through commands during operation, and monitor the miss and hit of the cache through the INFO command for tuning.

Internal implementation of elimination strategy

• The client executes a command, which causes the data in Redis to increase and occupy more memory
• Redis checks the memory usage, if it exceeds the maxmemory limit, clear some keys according to the policy
• Continue to execute the next command, and so on

In this process, the memory usage will continue to reach the limit value, and then exceed, and then delete some keys, the usage drops below the limit value.

If a command causes a large amount of memory usage (such as saving a large set with a new key), within a period of time, the memory usage may significantly exceed the maxmemory limit.

The difference between Redis and Memcached

Both are non-relational memory key-value databases. Now companies generally use Redis to implement caching, and Redis itself is getting more and more powerful! The difference between Redis and Memcached, please refer to: The difference between Redis and Memcached

Affairs

Redis implements transaction functions through commands such as MULTI, EXEC, and WATCH. Transaction provides a mechanism to package multiple command requests, and then execute multiple commands one-time and sequentially. During the execution of the transaction, the server will not interrupt the transaction and execute the command requests of other clients instead. All the commands in the transaction are executed, and then the command requests from other clients are processed.

Multiple commands in a transaction are sent to the server at once, rather than one by one. This method is called pipelining, which can reduce the number of network communications between the client and the server and improve performance.

In traditional relational databases, ACID properties are often used to test the reliability and security of transaction functions. In Redis, transactions always have Atomicity, Consistency, and Isolation, and when Redis runs in a specific persistence mode, transactions also have Durability.

• event

The Redis server is an event-driven program.

• File event

The server communicates with the client or other servers through the socket, and the file event is an abstraction of the socket operation.

Redis has developed its own network event handler based on the Reactor model. It uses I/O multiplexing programs to monitor multiple sockets at the same time, and transmits the arriving events to the file event dispatcher, which will be based on the socket The generated event type calls the corresponding event handler.

• Time event

The server has some operations that need to be executed at a given point in time, and time events are an abstraction of such timing operations.

Time events are divided into:

• Timed event: is to allow a program to be executed once within a specified time
• Periodic event: a program is executed every specified time

Currently Redis only uses periodic events instead of timed events. An event time is mainly composed of three attributes:

• id: the globally unique ID created by the server for the time event
• when: UNIX timestamp with millisecond precision, which records the arrival time of time events
• timeProc: time event handler, a function

The implementation server puts all time events in an unordered linked list. Whenever the time event executor runs, it traverses the entire linked list, finds all the time events that have arrived, and calls the corresponding event handler. (The linked list is an unordered linked list, not sorted by the size of the when attribute)

• Event scheduling and execution

The server needs to continuously monitor the file event socket to get the pending file event, but it cannot always monitor, otherwise the time event cannot be executed within the specified time, so the listening time should be determined based on the closest time event to the present.

Sentinel

Sentinel can monitor the servers in the cluster and automatically elect a new master server from the slave servers when the master server goes offline.

Redis cluster production environment high-availability solution actual combat process

Fragmentation

Sharding is a method of dividing data into multiple parts, which can be stored in multiple machines. This method can achieve a linear level of performance improvement when solving certain problems.

Assuming there are 4 Redis instances R0, R1, R2, R3, and many keys representing users user:1, user:2, ..., there are different ways to choose which instance a specified key is stored in.

The simplest way is range fragmentation, for example, user IDs from 0 to 1000 are stored in instance R0, user IDs from 1001 to 2000 are stored in instance R1, and so on. However, a mapping range table needs to be maintained in this way, and the maintenance operation is costly.

Another way is to hash fragments, use the CRC32 hash function to convert the key into a number, and then modulo the number of instances to know the instances that should be stored.

According to the location where the fragmentation is performed, it can be divided into three fragmentation methods:

• Client fragmentation: The client uses algorithms such as consistent hashing to determine which node the key should be distributed to.
• Proxy fragmentation: The client request is sent to the proxy, and the proxy forwards the request to the correct node.
• Server fragmentation: Redis Cluster.

copy

Use the slaveof host port command to make one server a slave of another server.

A slave server can only have one master server, and master-master replication is not supported.

• Connection process

  • The master server creates a snapshot file, sends it to the slave server, and uses the buffer to record the executed write commands during the sending. After the snapshot file is sent, start to send the write command stored in the buffer to the slave server
  • The slave server discards all old data, loads the snapshot file sent by the master server, and then the slave server starts to accept write commands sent by the master server
  • Each time the master server executes a write command, it sends the same write command to the slave server
• Master-slave chain

As the load continues to rise, the master server may not be able to update all the slave servers quickly, or reconnecting and resynchronizing the slave servers will cause the system to overload. In order to solve this problem, an intermediate layer can be created to share the replication work of the primary server. The middle-tier server is the slave server of the top-tier server, and the master server of the bottom-tier server.

Solutions to problems such as cache avalanche and cache penetration in Redis

Cache avalanche

Cache avalanche refers to a large area of cache failure at the same time, so subsequent requests will fall on the database, causing the database to withstand a large number of requests in a short period of time and crash.

solution

• The expiration time of cached data is set randomly to prevent a large amount of data from expiring at the same time.
• Generally, when the amount of concurrency is not particularly high, the most used solution is to lock and queue.
• Add a corresponding cache mark to each cached data, record whether the cache is invalid, if the cache mark is invalid, update the data cache.

Cache penetration

Cache penetration refers to data that is not in the cache or in the database, causing all requests to fall on the database, causing the database to crash due to a large number of requests in a short period of time.

solution

• The interface layer adds verification, such as user authentication verification, id is used for basic verification, and id<=0 is directly intercepted;
• The data that cannot be retrieved from the cache is also not retrieved in the database. At this time, the key-value pair can also be written as key-null, and the effective time of the cache can be set short, such as 30 seconds (setting too long will cause normal conditions Can not be used). This can prevent attacking users from repeatedly using the same id brute force attack
• Using bloom filters, hash all possible data into a bitmap that is large enough, and a data that must not exist will be intercepted by this bitmap, thereby avoiding the query pressure on the underlying storage system

Attach

• The use of space has reached an extreme, that is, Bitmap and Bloom Filter.
• Bitmap: A typical hash table
• The disadvantage is that Bitmap can only record 1bit information for each element. If you want to complete additional functions, I am afraid that you can only do it by sacrificing more space and time.

Bloom filter (recommended)

• It is the introduction of k(k>1)k(k>1) mutually independent hash functions to ensure that the process of determining the weight of the element is completed under a given space and misjudgment rate.
• Its advantage is that the space efficiency and query time far exceed the general algorithm, but the disadvantage is that it has a certain misrecognition rate and difficulty in deletion.
• The core idea of Bloom-Filter algorithm is to use multiple different Hash functions to resolve "conflicts".
• Hash has a conflict (collision) problem, and the values of two URLs obtained with the same Hash may be the same. In order to reduce conflicts, we can introduce a few more hashes. If we find that an element is not in the set through one of the hash values, then the element is definitely not in the set. Only when all the Hash functions tell us that the element is in the set, can it be determined that the element exists in the set. This is the basic idea of Bloom-Filter.
• Bloom-Filter is generally used to determine whether an element exists in a large data set.

Cache breakdown

Cache breakdown refers to the data that is not in the cache but in the database (usually when the cache time expires). At this time, because there are so many concurrent users, and the data is not read in the read cache at the same time, the data is retrieved from the database at the same time, causing database pressure Increase instantly, causing excessive pressure. Unlike cache avalanche, cache breakdown refers to concurrently checking the same piece of data. Cache avalanche means that different data has expired, and many data cannot be found, so you can check the database.

solution

• Set hotspot data to never expire.
• Add mutex lock

Cache warm-up

Cache warm-up is to directly load related cache data into the cache system after the system is online. In this way, you can avoid the problem of querying the database first and then caching the data when the user requests it! The user directly queries the pre-heated cache data!

solution

• Write a cache to refresh the page directly, and do it manually when you go online;
• The amount of data is not large, and it can be loaded automatically when the project is started;
• Refresh the cache regularly;

Cache degradation

When the traffic increases sharply, the service has problems (such as slow response time or unresponsive), or non-core services affect the performance of the core process, it is still necessary to ensure that the service is still available, even if it is detrimental to the service. The system can automatically degrade based on some key data, or it can be configured with switches to achieve manual degradation.

The ultimate goal of cache degradation is to ensure that core services are available, even if they are lossy. And some services cannot be downgraded (such as adding to shopping cart, settlement).

Before downgrading, you need to sort out the system to see if the system can lose the pawn and protect the commander; thus sort out which must be protected and which can be downgraded; for example, you can refer to the log level setting plan:

General: For example, some services occasionally time out due to network jitter or the service is online, and they can be automatically degraded;

Warning: Some services fluctuate in the success rate within a period of time (for example, between 95% and 100%), and they can be automatically or manually degraded, and an alarm will be sent;

Error: For example, the availability rate is lower than 90%, or the database connection pool is burst, or the traffic suddenly increases to the maximum threshold that the system can withstand. At this time, it can be automatically downgraded or manually downgraded according to the situation;

Serious error: For example, the data is wrong due to special reasons, and urgent manual downgrade is required at this time.

The purpose of service downgrading is to prevent Redis service failures, resulting in an avalanche problem in the database. Therefore, for unimportant cached data, a service degradation strategy can be adopted. For example, a common practice is that Redis does not query the database, but directly returns the default value to the user.

Hot data and cold data

Hot data, cache is valuable

For cold data, most of the data may have been squeezed out of memory before being accessed again, which not only takes up memory, but also has little value. Frequently modified data, consider using cache depending on the situation

For hot data, such as one of our IM products, birthday wishes module, and the birthday list of the day, the cache may be read hundreds of thousands of times in the future. For another example, for a navigation product, we cache the navigation information, which may be read millions of times in the future.

The data is read at least twice before the update, and the cache is meaningful. This is the most basic strategy. If the cache fails before it works, it won't be of much value.

Does it exist, the frequency of modification is high, but the cache has to be considered? Have! For example, this reading interface puts a lot of pressure on the database, but it is also hot data. At this time, we need to consider caching to reduce the pressure on the database, such as our assistant product, the number of likes, the number of favorites, and the number of shares. Waiting is a very typical hot data, but it is constantly changing. At this time, it is necessary to synchronize the data to the Redis cache to reduce database pressure.

Cache hot key

A Key in the cache (such as a promotional item), when it expires at a certain point in time, there are a large number of concurrent requests for this Key at this point in time. These requests will generally load data from the back-end DB when the cache expires. Set back to the cache. At this time, a large concurrent request may instantly overwhelm the back-end DB.

solution

Lock the cache query, if the KEY does not exist, lock it, then check the DB into the cache, and then unlock; if other processes find a lock, they wait, and then wait for the unlock to return the data or enter the DB query

Reference link for the above content: https://blog.csdn.net/ThinkWon/article/details/99999584

Redis optimization best practices

Let's summarize Redis , which mainly includes two levels: business level and operation and maintenance level.

Since I have written a lot of UGC back-end services before, and used Redis in a large number of scenarios, I have also stepped on a lot of pits in this process, so I also summarized a set of reasonable usage methods during the use process.

Later, I developed the infrastructure and developed Codis, Redis related middleware. At this stage, the focus of the field is sinking from the use level to the development and operation of Redis, and more focus is on the internal implementation and operation and maintenance of Redis. This kind of problem, I have also accumulated some experience in this area.

For these two areas, I will share with you what I think is a reasonable way to use Redis and operation and maintenance. It may not be the most comprehensive, and it may be different from the way you use Redis, but the following methods are the actual conclusions I have summarized after stepping on the pit. Experience, for your reference. Follow the public number Java technology stack and reply to redis to get a series of Redis tutorials.

At the business level, developers need to pay attention, that is, how to use Redis reasonably when writing business code. Developers need to Redis in order to use Redis in appropriate business scenarios, thereby avoiding business-level delay problems.

During the development process, the optimization suggestions at the business level are as follows:

• The length of the key should be as short as possible. When the amount of data is very large, a long key name will occupy more memory
• Be sure to avoid storing excessively large data (large value). Excessively large data consumes a lot of time when allocating and releasing memory, and will block the main thread.
• For Redis 4.0 or higher, it is recommended to enable the lazy-free mechanism to operate asynchronously when large values are released, without blocking the main thread
• It is recommended to set the expiration time and use Redis as a cache, especially when the number is large, not setting the expiration time will cause unlimited memory growth
• Do not use highly complex commands, such as SORT, SINTER, SINTERSTORE, ZUNIONSTORE, ZINTERSTORE. Using these commands takes a long time and will block the main thread
• When querying data, try to get as little data as possible at a time. When you are not sure about the number of container elements, avoid operations such as LRANGE key 0 -1 and ZRANGE key 0 -1. You should set the number of specific query elements. It is recommended Query less than 100 elements at a time
• When writing data, try to write as little data as possible at a time, such as HSET key value1 value2 value3...,-control the number of elements written at one time, it is recommended to be less than 100, and large data volumes are written in multiple batches
• When operating data in batches, replace GET/SET with MGET/MSET, and replace HGET/HSET with HMGET/MHSET to reduce the number of network IO requests back and forth and reduce latency. For commands without batch operations, it is recommended to use pipeline to send multiple commands at once To the server
• It is forbidden to use the KEYS command. When you need to scan the instance, it is recommended to use SCAN. The online operation must control the scanning frequency to avoid performance jitter to Redis.
• To avoid a large number of keys expiring at a certain point in time, it is recommended to add a random time when the expiration is concentrated to break up the expiration time, reduce the pressure of Redis when the key expires in the concentration, and avoid blocking the main thread
• According to the business scenario, choose the appropriate elimination strategy, usually the random expiration is faster than the LRU expiration to eliminate the data
• Use the connection pool to access Redis, and configure reasonable connection pool parameters to avoid short connections. The TCP three-way handshake and four wave of hands are also time-consuming
• Only use db0. Multiple dbs are not recommended. Using multiple dbs will increase the burden on Redis. Each time you access a different db, you need to execute the SELECT command. If the business lines are different, it is recommended to split multiple instances and increase a single instance. Performance
• When the number of read requests is large, it is recommended to use read-write separation, provided that the problem of not updating the data from the section in time can be tolerated
• When the amount of write requests is large, it is recommended to use a cluster and deploy multiple instances to share the write pressure

Operational level

The purpose is to reasonably plan the deployment of Redis and ensure the stable operation of Redis. The main optimizations are as follows:

• Deploy different instances of different business lines, independent of each other to avoid mixing. It is recommended that different business lines use different machines, and deploy them in different groups according to the importance of the business to prevent problems in one business line from affecting other business lines
• Ensure that the machine has sufficient CPU, memory, bandwidth, and disk resources to prevent excessive load from affecting Redis performance
• Deploy instances in the master-slave cluster mode and distribute them on different machines to avoid single points. The slave must be set to readonly
• The machines where the master and slave nodes are located are independent of each other. Do not cross-deploy instances. Normally, the backup work will be done on the slave, which will consume machine resources when doing the backup. Cross-deployment will affect the performance of the master
• It is recommended to deploy sentinel nodes to increase availability. The number of nodes should be at least 3, and they should be distributed on different machines to realize automatic failover.
• Plan the capacity in advance. The upper limit of the memory of a machine deployment instance is preferably half of the machine's memory. The master-slave synchronization will occupy up to an extra double the memory space to prevent large-scale network failures from causing the full amount of all master-slaves. Synchronization causes the machine's memory to be eaten up
• Do a good job of monitoring the CPU, memory, bandwidth, and disk of the machine, and promptly alarm when resources are insufficient. After Redis uses Swap, the performance drops sharply, the network bandwidth load is too high, and the access delay is significantly increased. When the disk IO is too high, turning on AOF will slow down Redis performance
• Set the upper limit of the maximum number of connections to prevent excessive client connections from causing excessive service load
• The memory usage of a single instance is recommended to be controlled below 20G. An excessively large instance will lead to longer backup time, high resource consumption, and longer block time for master-slave data synchronization.
• Set a reasonable slowlog threshold, 10 milliseconds are recommended, and monitor it. If too many slow logs are generated, timely alarm is required. Set a reasonable size of the replication buffer repl-backlog. Properly increasing the repl-backlog can reduce the probability of master-slave full replication.
• Set a reasonable client-output-buffer-limit size of the slave node. For instances with a large amount of writes, properly adjust the size to avoid the interruption of master-slave replication.
• It is recommended to do the backup on the slave node, which does not affect the performance of the master
• Do not enable AOF or enable AOF to configure to flush disk every second to avoid disk IO consumption and reduce Redis performance
• When the instance has set the upper limit of memory and needs to increase the upper limit of memory, adjust the slave first and then adjust the master, otherwise the data of the master and slave nodes will be inconsistent
• Add monitoring to Redis. When monitoring and collecting info information, use long connections. Frequent short connections will also affect Redis performance. redis performance monitoring index , refer to this article
• When scanning the entire number of instances online, remember to set the sleep time to avoid a sudden increase in QPS during scanning to cause performance jitters on Redis
• Do a good job in the runtime monitoring of Redis, especially the expired\_keys, evicted\_keys, latest\_fork\_usec indicators. A sudden increase in the value of these indicators in a short period of time may block the entire instance and cause performance problems

The above is the practice method recommended by Redis when I use Redis and develop Redis-related middleware. These aspects mentioned above have been encountered more or less in actual use.

It can be seen that in order to stably exert the high performance of Redis, we need to do a good job in all aspects, but any problem in one aspect will inevitably affect the performance of Redis, which puts forward higher requirements for our use and operation and maintenance.

If you encounter more problems or have better experience in using Redis, you can leave a message and discuss it together!

Source: kaito-kidd.com/2020/07/04/redis-best-practices/

Redis other summary

Cloud Redis Development Specification Study Summary

The most comprehensive horizontal evaluation of Redis visualization tools at present

Redis performance test tool introduction

Redis visual management tool

Redis is getting slower and slower? Location and analysis of common delay problems

Redis distributed cluster deployment

principle:

• The Redis cluster uses a consistent hash slot method to allocate a certain hash slot to each master node in the cluster, and hash the written data and assign it to a master node for storage.
• The cluster uses the formula (CRC16 key) & 16384 to calculate the slot of the key data.
• 16,384 slots are evenly distributed on each node.
• Each master node in the cluster will undertake part of the maintenance of the slots, and the slots store data, and each master node has at least one slave node for high availability.

Node communication method:

• Open a port number +10000, which is used to communicate and exchange information between nodes in the cluster.
• By default, each node selects 5 random nodes to send ping messages 10 times per second, and transmits its own information and known cluster information. After receiving the ping message, it returns a pong message as a reply. Finally, through this random message exchange, each The node will get all the information.
• When a master node goes down, all nodes will find that the master node is down. As the slave node of the master node, it will take over the work of the master node, and then tell all other nodes that he has become the master. In this way, other surviving nodes will update the information table they maintain, and the slave node will take over as the master, and an error will be reported if all the clusters are down. When operating from a node, it will be stored in one of the master nodes based on the consistent hash calculation, and the slave nodes will synchronize the data of the master.

• The redis cluster is decentralized. Each node in the cluster has an equal relationship, and each node saves its own data and the state of the entire cluster. Each node is connected to all other nodes, and these connections remain active.
• When building a cluster, each shard's master node corresponds to a slave node. Realize the slaveof function, and when the master node is down, realize the automatic failover switching function of sentinel

Redis distributed cluster (deployment):

Port number: 7000-7005

This time, the distributed sharded cluster can be on one Linux system, and only multiple instances need to be installed as a cluster configuration.

Install ruby environment support:

yum -y install ruby rubygems

Yum installs version 2.0.0, but gem needs version 2.2.2 or higher, so choose to compile

Download and install ruby environment:

wget https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.1.tar.gz
tar xf ruby-2.6.1.tar.gz && cd ruby-2.6.1/
./configure --prefix=/usr/local/ruby
make && make install && echo $?
echo 'export PATH=$PATH:/usr/local/ruby/bin' >> /etc/profile
source /etc/profile

Modify the domestic source of gem tools:

# 查看gem工具源地址
gem sources -l
# 添加一个阿里云的gem工具源
gem sources -a http://mirrors.aliyun.com/rubygems/
# 删除gem工具默认国外源
gem sources --remove https://rubygems.org/
# 下载当前最新版本集群插件
gem install redis -v 4.1.0

Cluster node preparation:

mkdir /data/700{0..5}

Example of configuring port 7000:

vim /data/7000/redis.conf
port 7000
daemonize yes
pidfile /data/7000/redis.pid
loglevel notice
logfile "/data/7000/redis.log"
dbfilename dump.rdb
dir /data/7000
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes

Copy other port instances:

# 拷贝配置文件
cp /data/7000/redis.conf /data/7001/
cp /data/7000/redis.conf /data/7002/
cp /data/7000/redis.conf /data/7003/
cp /data/7000/redis.conf /data/7004/
cp /data/7000/redis.conf /data/7005/

# 修改配置文件内容
sed -i 's#7000#7001#g' /data/7001/redis.conf
sed -i 's#7000#7002#g' /data/7002/redis.conf
sed -i 's#7000#7003#g' /data/7003/redis.conf
sed -i 's#7000#7004#g' /data/7004/redis.conf
sed -i 's#7000#7005#g' /data/7005/redis.conf

Start all instances:

redis-server /data/7000/redis.conf
redis-server /data/7001/redis.conf
redis-server /data/7002/redis.conf
redis-server /data/7003/redis.conf
redis-server /data/7004/redis.conf
redis-server /data/7005/redis.conf

Create command soft link:

(This command has expired, now use the redis-cli command) (optional execution command)

ln -s /usr/local/redis-5.0.2/src/redis-trib.rb /usr/sbin/

View the process:

ps -ef |grep redis-server

Add all instance nodes to the cluster management:

# --replicas 1",1是代表每一个主有一个从,后面的是所有节点的地址与端口信息
redis-cli --cluster create --cluster-replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

The distributed master-slave rule is that the first three instance nodes are the master, and the corresponding latter three instance nodes are the slave nodes. If replicas 2, then add 3 more instance nodes

View the status of the master node:

redis-cli -p 7000 cluster nodes|grep master

View the status of the slave node:

redis-cli -p 7000 cluster nodes|grep slave

Redis-distributed cluster (management)

Preparation for adding cluster nodes:

mkdir /data/700{6..7}

拷贝其他端口实例：

# 拷贝配置文件
cp /data/7000/redis.conf /data/7006/
cp /data/7000/redis.conf /data/7007/

# 修改配置文件内容
sed -i 's#7000#7006#g' /data/7006/redis.conf
sed -i 's#7000#7007#g' /data/7007/redis.conf

Start the new node instance:

redis-server /data/7006/redis.conf
redis-server /data/7007/redis.conf

View the process:

ps -ef |grep redis-server

Add the master node: (7000 instance is the management node)

#'把7006实例添加到7000实例这个主节点所在集群内(此时已经有了4个主节点)
redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000

View the status of the master node:

redis-cli -p 7000 cluster nodes|grep master

Transfer slot (re-sharding):

#'操作集群管理节点从新分配，并在交互界面指定分片大小、选择接收分片的节点ID
redis-cli --cluster reshard 127.0.0.1:7000

How many slots do you want to move (from 1 to 16384)? 4096
#通过人工手动计算数据分片总大小除以主节点后的数字

What is the receiving node ID? 2129d28f0a86fc89571e49a59a0739812cff7953
#选择接收数据分片的节点ID，（这是新增节点7006实例的ID号）

Source node #1: all
#选择从哪些源主节点重新分片给新主节点）（all是所有节点）

Do you want to proceed with the proposed reshard plan (yes/no)? yes           
#确认修改以上的操作

Re-check the status of the master node: (you can see the re-sharding of the cluster data)

redis-cli -p 7000 cluster nodes|grep master

Add slave node:

#'把7007实例节点添加到7006实例主节点内，并指定对应7006实例主节点坐在集群的管理节点
redis-cli --cluster add-node 127.0.0.1:7007 127.0.0.1:7000 --cluster-slave --cluster-master-id 2129d28f0a86fc89571e49a59a0739812cff7953

View the status of the slave node:

redis-cli -p 7000 cluster nodes|grep slave

Cluster node deletion preparation:

Move the data fragment of the node to be deleted:

#'操作集群管理节点从新分配，并在交互界面指定分片大小、选择接收分片的节点ID
redis-cli --cluster reshard 127.0.0.1:7000

#方法是根据要删除master节点的分片位置，然后一个组分一个节点 ， 也可以直接移动所有数据片到一个节点

How many slots do you want to move (from 1 to 16384)? 1365                    
#通过人工手动查看数据分片总大小

What is the receiving node ID? e64f9074a3733fff7baa9a4848190e56831d5447
#选择接收数据分片的节点ID，（这是新增节点7006实例的ID号）

Source node #1: 2129d28f0a86fc89571e49a59a0739812cff7953
#选择从哪些源主节点重新分片给新主节点（这是要删除的主节点的ID号）

Source node #2: done
#这是结束命令

Do you want to proceed with the proposed reshard plan (yes/no)? yes           
#确认修改以上的操作

Re-check the status of the master node: (you can see the re-sharding of the cluster data)

redis-cli -p 7000 cluster nodes|grep master

Continue to move the data piece:

#'操作集群管理节点从新分配，并在交互界面指定分片大小、选择接收分片的节点ID
redis-cli --cluster reshard 127.0.0.1:7000

# 方法是根据要删除master节点的分片位置，然后一个组分一个节点 ， 也可以直接移动所有数据片到一个节点

How many slots do you want to move (from 1 to 16384)? 1366                    #通过人工手动查看数据分片总大小

What is the receiving node ID? f6c1aaea3a8c56e0c7dee8ad7ae17e26dd04244c
#选择接收数据分片的节点ID，（这是新增节点7006实例的ID号）

Source node #1: 2129d28f0a86fc89571e49a59a0739812cff7953
#选择从哪些源主节点重新分片给新主节点（这是要删除的主节点的ID号）

Source node #2: done
#这是结束命令

Do you want to proceed with the proposed reshard plan (yes/no)? yes           
#确认修改以上的操作

Re-check the status of the master node: (you can see the re-sharding of the cluster data)

redis-cli -p 7000 cluster nodes|grep master

The last time to move the data piece:

#'操作集群管理节点从新分配，并在交互界面指定分片大小、选择接收分片的节点ID
redis-cli --cluster reshard 127.0.0.1:7000

#方法是根据要删除master节点的分片位置，然后一个组分一个节点 ， 也可以直接移动所有数据片到一个节点

How many slots do you want to move (from 1 to 16384)? 1365                    
#通过人工手动查看数据分片总大小

What is the receiving node ID? 5a0df4ea0af5f35c1248e45e88d44c3f2e10169f
Source node #1: 2129d28f0a86fc89571e49a59a0739812cff7953
Source node #2: done                

Re-check the status of the master node: (you can see the re-sharding of the cluster data)

redis-cli -p 7000 cluster nodes|grep master

Delete the master node that emptied the data slice:

#'删除已经清空数据的7006实例
redis-cli --cluster del-node 127.0.0.1:7006 2129d28f0a86fc89571e49a59a0739812cff7953

#删除没有主库的7007实例
redis-cli --cluster del-node 127.0.0.1:7007 821bcf78c5e4c0b08aa7a5d514214b657ebec4ab

Other configuration management:

#内存信息查看
redis-cli -p 6382 -a redhat info memory

#设置最大只能使用100MB的内存
redis-cli -p 6382 -a redhat config set maxmemory 102400000

most complete and latest Redis interview questions in history (latest version)