Memory optimization for Redis development and operation and maintenance

memory optimization

We all know that Redis data is stored in memory, and memory is a very valuable resource. This article will explain how to optimize memory.

redisObject object

First of all, you need to understand what redisObject is. All value objects stored in Redis are internally defined as redisObject , and the structure is as follows.

• type : Indicates the data type used by the current object, mainly string, hash, list, set, zset. 4 means occupy 4 bits.

  📢: Use the type [key] command to view the type of the object, the returned value is the type of the value object, and the keys are all string types.

• encoding : Indicates the type of internal encoding, which indicates which data structure is used by the current object. Understanding the internal encoding type is very important for memory optimization.

• lru : Record the time when the object was last accessed. When maxmemory and maxmemory-policy=volatile-lru or allkeys-lru are configured, it is used to assist the LRU algorithm to delete key data.

  📢: Use object idletime [key] to see the idle time of the current key without updating the lru field time.

  Use the scan + object idletime command to batch query those keys that have not been accessed for a long time and clean them up to reduce memory usage.

• refcount : Record the number of times the current object is referenced. When refcount=0, the object space can be reclaimed safely.

  📢: Use object recount [key] to get the current object reference. When the object value is in [0-9999], redis will use the shared object pool to save memory.

• * ptr : If the integer is stored, store the data directly, otherwise store the pointer to the data. When the value object is a string and <= 44 bytes, the internal encoding is embstr type, and when > 44 bytes, the raw type is used.

Reduce key-value objects

Reducing the length of key and value can effectively reduce Redis memory usage, such as abbreviating key.

The value is more complicated. If the business object is serialized into a binary array, unnecessary properties can be removed. Secondly, more efficient serialization tools such as protostuff, kryo, etc. can be selected. In addition to binary arrays, we also store strings such as json and xml. In the case of tight memory, we can use compression algorithms to compress json and xml and store them in redis.

shared object pool

Shared object pool refers to the integer object pool [0-9999] maintained in Redis. Because of the memory overhead of creating a large number of integer type redisObjects, a redisObject occupies at least 16 bytes internally, so Redis maintains an integer object pool to save memory. In addition, shared object pools can also be used in list, hash, set, and zset.

As shown in the figure above, you can see that when the value is 100, the refcount is 2147483647, which is actually INT_MAX, which is a shared object. And 12000 has a reference count of 1, which is a newly created object.

The redisObject at this time is as follows:

📢: It should be noted that when maxmemory and LRU-related elimination strategies are set: volatile-lru, allkeys-lru, Redis will disable the shared object pool at this time.

The LRU algorithm needs to obtain the last access time of the object, but there may be multiple references in the shared object pool pointing to the same redisObject at the same time. At this time, the lru field will also be shared, resulting in the inability to obtain the last access time of each object. But if maxmemory is not set, the recycling mechanism will not be triggered until the memory is used up, so the shared object pool can be used all the time.

📢: Another thing to note is that if the internal encoding uses ziplist value objects, the shared object pool cannot be used even if all data are integers. Because ziplist uses a compressed and memory-contiguous structure, object judgment is expensive.

String optimization

The most common in Redis is the string, all keys are of string type, and the value object data type is of string type except for integers. So how to perform string optimization is also one of the key points.

First, let's understand the structure of strings.

string structure

Redis does not use strings in C language, but implements the string structure by itself.

简单动态字符串（simple dynamic string）SDS

 struct sdshdr{
     //字节数组
     char buf[];
     //buf数组中已使用字节数量
     int len;
     //buf数组中未使用字节数量
     int free;
}

Advantages of SDS:

• O(1) time complexity to get the string length, used length, unused length.
• Byte arrays can be saved, enabling secure binary data storage.
• Internally implements a space pre-allocation mechanism to reduce the number of memory reallocations.
• Lazy deletion mechanism, the space is not released immediately after the string is reduced, but is reserved as pre-allocated space.

PS: For the relevant content of the string SDS, you can see the previous article.

📢: It should be noted that the pre-allocation mechanism for strings is used to prevent frequent modification of the string content resulting in frequent memory reallocation and string copying. Therefore, it is necessary to reduce modifications as much as possible, such as append, setrange. Instead, you can directly use set to modify the string to reduce memory waste and memory fragmentation caused by allocation.

String reconstruction

If you save json data, you can use the hash structure to store it, and use hmget and hmset to obtain and modify it in batches.

Test performance in the presence of long strings:

需要注意的是如果值对象字符串长度大于65则Redis会使用hashtable编码方式，反而会消耗更多内存。

By adjusting hash-max-ziplist-value=xx to set an appropriate value, the ziplist encoding method will be used, which will save more memory.

Coding optimization

Learn about encoding

Redis provides types such as string, list, hash, set, zet, etc. But there is a concept of different encoding for each type, which is actually which data structure is used at the bottom. Different encodings will directly affect memory usage and read and write efficiency.

Use the object encoding [key] command to get the encoding type.

 > object encoding jack
int
> hset hello student jack
1
> object encoding hello
ziplist
...

type	Encoding	data structure
string	raw	dynamic string
string	embstr	String to optimize memory allocation
string	int	integer
hash	hashtable	hash table
hash	ziplist	Compressed list
list	linkedlist	Doubly linked list
list	ziplist	Compressed list
list	quicklist	quick list
set	hashtable	hash table
set	intset	collection of integers
zset	skiplist	jump table
zset	ziplist	Compressed list

📢: The encoding type is automatically completed when Redis writes data. It can only be converted from small memory encoding to large memory encoding. The process is irreversible.

Next, let's see what the transition conditions look like.

type	coding	transition condition
string	embstr	value byte length <= 44
string	raw	value byte length > 44
string	int	integer
hash	ziplist	value byte length <=hash-max-ziplist-value and number of fields <=hash-max-ziplist-entries
hash	hashtable	value byte length>hash-max-ziplist-value or number of fields>hash-max-ziplist-entries
list	ziplist	value byte length <=list-max-ziplist-value and linked list length <=list-max-ziplist-entries
list	linkedlist	value byte length>list-max-ziplist-value or linked list length>list-max-ziplist-entries
list	quicklist	Discard the above list-max-ziplist-value and list-max-ziplist-entries configurations. Use: list-max-ziplist-size for maximum compression space or length. The maximum space is configured in the range of [-5~1], the default -2 means 8KB. A positive integer indicates the maximum compressed length. list-compress-depth: indicates the maximum compression depth, the default is 0 for no compression
set	intset	elements are integers and set length <=hash-max-ziplist-entries
set	hashtable	Element is non-integer or set length > hash-max-ziplist-entries
zset	ziplist	value byte length <=zset-max-ziplist-value and set length <=zset-max-ziplist-entries
zset	skiplist	value byte length>zset-max-ziplist-value or set length>zset-max-ziplist-entries

ziplist encoding

This article focuses on the introduction of ziplist. All data in ziplist is a memory structure stored in a linear continuous manner, which can be implemented as the underlying data structure of list, hash, and zset.

• zlbytes : Record the length in bytes occupied by the entire compressed list. The type is int-32 and the length is 4 bytes.
• zltail : record the offset from the tail node to facilitate the tail node pop-up operation. The type is int-32 and the length is 4 bytes.
• zllen : Record the number of compressed linked list nodes.
• entry : Record the specific node.
• prev_entry_bytes_length : Record the space occupied by the previous node, which is used to quickly locate the previous node, and can also implement reverse iteration of the list.
• encoding : indicates the current node code and length, the first two bits indicate the encoding type: string/integer, and the remaining bits indicate the data length.
• contents : holds the value of the node.
• zlend : The end of the record list, occupying one character.

Features:

• A contiguous memory array in which data is compactly arranged.
• Doubly linked list structure can be simulated, with O(1) time complexity for enqueuing and dequeuing.
• ziplist is extremely efficient in space utilization, and each entry is only a waste of 6 bytes at most.
• ziplist The underlying structure has no linked list, and the next or last node position is obtained through the memory offset
• ziplist has a high probability of chain update when inserting and deleting, so try to ensure that the number of stored value digits is the same when using it, otherwise the worst time complexity will be O(n^2).

Summarize

• Redis memory consumption mainly lies in: key-value objects, buffer memory.
• The maximum available memory of Redis is controlled by maxmemory . When the set memory size is exceeded, the memory recycling strategy is controlled according to maxmemory-policy .
• Small integer objects are optimized using a shared object pool.
• Prefer integers to save space over strings.
• Optimize string usage to avoid memory waste caused by preallocation.
• Optimize hash, list, zset structures using ziplist compression encoding.
• Use intset encoding to optimize integer sets.