Redis is very awkward, it will be spoiled if you don't understand the usage specifications

This may be the most relevant Redis usage specification

Brother Ma, I was criticized by the company leader yesterday.

I work for a single matchmaker Internet company, and launched an event to send my girlfriend to my girlfriend when I place an order on Double Eleven.

Who ever thought that after 12 o'clock in the morning, the number of users soared, and there was a technical failure that prevented users from placing orders, and the boss was furious at that time!

After searching, it was found that Redis reported Could not get a resource from the pool .

Connection resources are not available, and the connection volume of a single Redis in the cluster is very high.

As a result, various changes to the maximum number of connections and the number of waiting connections, although the frequency of error messages have been eased, but continues to report errors .

Later, after offline testing, it was found that the Redis was very large, and the average return data 1s.

Brother Code, can you share the specifications for using Redis? I want to be a real man who can't break fast!

Why is Redis through this article, we know that Redis is struggling for high performance and memory saving.

Therefore, only the standardized use of Redis can achieve high performance and save memory, otherwise Redis will not be able to withstand our fuss.

The Redis usage specification revolves around the following latitudes:

• Key-value pair usage specification;
• Command usage specification;
• Data storage specifications;
• Operation and maintenance specifications.

Key-value pair usage specification

There are two points to note:

1. A good key can provide a key with strong readability and high maintainability, which is convenient for locating problems and finding data.
2. value should avoid bigkey , choose efficient serialization and compression, use object sharing pool, choose efficient and appropriate data type (refer to " Redis Practical Chapter: Using Data Types to Achieve Billion Statistics ").

key naming convention

The standardized key naming can be easily located when problems are encountered. Redis belongs to the NoSQL database Scheme

Scheme on specifications to establish its 061a88b21a72bc semantics is like building different databases according to different scenarios.

Knock on the blackboard

Use "business module name" as a prefix (for example, database Scheme ), separated by "colon", and add "specific business name".

In this way, we can key , which is clear.

In summary: "Business name: table name: id"

For example, we want to count the number of fans of the technical blogger "Megabytes" whose official account is a technical type.

set 公众号:技术类:码哥字节 100000

Brother Ma, is there any problem if the key is too long?

The key is a string, and the underlying data structure is SDS . The SDS structure will contain metadata information such as the length of the string and the size of the allocated space.

string length of 161a88b21a73db increases, SDS metadata will also take up more memory space.

So when the string is too long, we can use appropriate abbreviations.

Don't use bigkey

Brother Ma, I was slapped, and an error was reported and the connection could not be obtained.

Because Redis executes read and write instructions in a single thread, if bigkey occur, the thread will be blocked and the processing efficiency of Redis will be reduced.

bigkey contains two situations:

• Key-value pairs value large, such as value saved 2MB of String data;
• value of the key-value pair is a collection type with many elements, such as the List collection that stores 50,000 elements.

Although Redis officially stated that key and string type value restrictions are both 512MB .

prevents network card traffic and slow queries. The string is controlled within 10KB , and the hash、list、set、zset elements should not exceed 5000.

Brother Ma, what if the business data is so big? For example, the masterpiece "Jin Ping Mei" is preserved.

We can also reduce the data size gzip

/**
 * 使用gzip压缩字符串
 */
public static String compress(String str) {
    if (str == null || str.length() == 0) {
        return str;
    }

    try (ByteArrayOutputStream out = new ByteArrayOutputStream();
    GZIPOutputStream gzip = new GZIPOutputStream(out)) {
        gzip.write(str.getBytes());
    } catch (IOException e) {
        e.printStackTrace();
    }
    return new sun.misc.BASE64Encoder().encode(out.toByteArray());
}

/**
 * 使用gzip解压缩
 */
public static String uncompress(String compressedStr) {
    if (compressedStr == null || compressedStr.length() == 0) {
        return compressedStr;
    }
    byte[] compressed = new sun.misc.BASE64Decoder().decodeBuffer(compressedStr);;
    String decompressed = null;
    try (ByteArrayOutputStream out = new ByteArrayOutputStream();
    ByteArrayInputStream in = new ByteArrayInputStream(compressed);
    GZIPInputStream ginzip = new GZIPInputStream(in);) {
        byte[] buffer = new byte[1024];
        int offset = -1;
        while ((offset = ginzip.read(buffer)) != -1) {
            out.write(buffer, 0, offset);
        }
        decompressed = out.toString();
    } catch (IOException e) {
        e.printStackTrace();
    }
    return decompressed;
}

Collection type

If there are indeed many elements of the collection type, we can split a large collection into multiple small collections to save.

Use efficient serialization and compression methods

In order to save memory, we can use efficient serialization methods and compression methods to reduce the size of value

protostuff serialization methods 061a88b21a762a and kryo are more efficient than the serialization method built in Java

Although the above two serialization methods save memory, they are all binary data after serialization, which is too readable.

Usually we will serialize it to JSON or XML . In order to avoid large data occupation, we can use compression tools (snappy, gzip) to compress the data and save it in Redis.

Use integer object shared pool

Redis maintains 10,000 integer objects from 0 to 9999 internally, and uses these integers as a shared pool.

Even if a large number of key-value pairs save integers in the range of 0 to 9999, in the Redis instance, only one integer object is saved, which can save memory space.

It should be noted that there are two situations that do not take effect:

1. maxmemory is set in Redis, and the LRU strategy ( allkeys-lru or volatile-lru strategy) is enabled, then the integer object shared pool cannot be used.

   This is because LRU needs to count the usage time of each key-value pair. If different key-value pairs reuse an integer object, the statistics cannot be counted.

2. If the collection type data is encoded by ziplist and the collection elements are integers, the shared pool cannot be used at this time.

   Because ziplist uses a compact memory structure, it is inefficient to determine the sharing of integer objects.

Command usage specification

The execution of some commands will cause big performance problems, we need to pay extra attention.

Production prohibited instructions

Redis is a single-threaded processing request operation. If we execute some commands that involve a large number of operations and take a long time, it will severely block the main thread and cause other requests to not be processed normally.

• KEYS: This command requires a full table scan of Redis's global hash table, which seriously blocks the Redis main thread;

  SCAN should be used instead, returning qualified key-value pairs in batches to avoid blocking the main thread.

• FLUSHALL: Delete all data on the Redis instance. If the amount of data is large, it will seriously block the Redis main thread;

• FLUSHDB, delete the data in the current database, if the amount of data is large, it will also block the Redis main thread.

  Add the ASYNC option to let FLUSHALL and FLUSHDB execute asynchronously.

We can also disable it directly and use the rename-command command to rename these commands in the configuration file so that the client cannot use these commands.

Use the MONITOR command with caution

The MONITOR command will continue to write the monitored content into the output buffer.

If there are many operations on online commands, the output buffer will soon overflow, which will affect Redis performance and even cause service crashes.

Therefore, unless it is necessary to monitor the execution of certain commands (for example, Redis performance suddenly slows down, we want to see which commands the client executes) we use it.

Use full operation commands with caution

For example, get all the elements in the set (hgetall of HASH type, lrange of List type, smembers of Set type, zrange, etc.).

These operations will perform a full scan of the entire underlying data structure, resulting in blocking the Redis main thread.

Brother Code, what if the business scenario just needs to get the full amount of data?

There are two ways to solve it:

1. Use SSCAN、HSCAN and other commands to return the collection data in batches;
2. Divide large collections into small collections, such as by time, area, etc.

Data preservation specification

Separate cold and hot data

Although Redis supports the use of RDB snapshots and AOF logs for persistent storage of data, these two mechanisms are used to provide data reliability guarantees, and are not used to expand data capacity.

Do not store all data in Redis, it should be used as a cache to store hot data , so that you can make full use of the high-performance features of Redis, and you can also use precious memory resources to serve hot data.

Business data isolation

Don't put all irrelevant data services into one Redis. On the one hand, it avoids the mutual influence of business, on the other hand, it avoids the expansion of single instance, and can reduce the impact area and quickly recover in the event of a failure.

Set expiration time

When saving data, I suggest that you set the expiration time of the data according to the length of time the business uses the data.

Data written to Redis will always occupy memory. If the data continues to increase, it may reach the upper limit of the machine's memory, causing memory overflow and service crashes.

Control the memory capacity of a single instance

The recommended setting is 2~6 GB. In this way, whether it is RDB snapshots or master-slave cluster data synchronization, it can be completed quickly without blocking the processing of normal requests.

Prevent cache avalanche

Avoid centralizing expired keys and causing cache avalanches.

Brother Code, what is a cache avalanche?

When a large-scale cache failure occurs at a certain moment, it will cause a large number of requests to be directly hit on the database, resulting in huge database pressure. If it is high concurrency, it may cause the database to go down in an instant.

Operation and maintenance specifications

1. Use Cluster cluster or sentinel cluster to achieve high availability;
2. The instance sets the maximum number of connections to prevent too many client connections from causing excessive load on the instance and affecting performance.
3. Do not enable AOF or enable AOF to configure to flush disks every second to avoid disk IO slowing down Redis performance.
4. Set a reasonable repl-backlog to reduce the probability of master-slave full synchronization
5. Set a reasonable slave client-output-buffer-limit to avoid interruption of master-slave replication.
6. Set an appropriate memory elimination strategy according to the actual scenario.
7. Use connection pool to operate Redis.

Finally, welcome to share your common usage specifications in the message area, and we will discuss together.

good article recommendation

Redis Core: Why is it so fast

Redis Persistence: How AOF and RDB ensure high data availability

Redis High Availability: Master-Slave Architecture Data Consistency and Synchronization Principle

Redis High Availability: Principles of Sentinel Cluster

Redis High Availability: Cluster Principle

Redis Actual Combat: Using Bitmap

Redis Actual Combat: Realize nearby people

Redis New Features: Interpretation of the Multithreading Model

New Features of Redis 6.0: The Revolution