[Redis series] redis learning eight, redis persistent RDB and AOF

Redis persistence

Redis is an in-memory database. If the in-memory database is not saved to disk, once the server goes down or the redis process exits, not only the data will be lost, but the database state in the server will also be lost.

So redis provides the function of persistence

The persistence of redis is divided into RDB and AOF

RDB (Redis DatabBase)

In master-slave replication, rdb files are used as backup and placed on the slave machine

The snapshot of the data set in memory is written to the disk within the specified time interval, which is the snapshot snapshot. When restoring the snapshot, the snapshot file is read into the memory.

Redis creates a child process by fork to do persistent actions.

• First write the data to a temporary file, and when the persistence process is over, replace the last persistent file with this temporary file

During the whole process, the main process does not perform task IO operations, which ensures extremely high performance

If large-scale data recovery is required, and the requirements for data integrity are not so sensitive and strict, the RDB persistence method is better and more efficient than the AOF persistence method.

Although RDB has high performance, the data may be lost after the last persistence. Redis uses the RDB persistence method by default, and generally does not need to be modified.

 save 60 3 # The filename where to dump the DB dbfilename dump.rdb dir ./

If we set redis to operate 3 times within 60s, then do a persistence

 127.0.0.1:6379> ping PONG 127.0.0.1:6379> config get dir 1) "dir" 2) "/root" 127.0.0.1:6379> set k1 v1 OK 127.0.0.1:6379> set k2 v2 OK 127.0.0.1:6379> set k3 v3 OK

The dump.db file is generated in the dir directory, our dir directory is /root

After performing the above operations, we found that the dump.rdb file was generated in /root, let's delete the file and try again

 root@iZuf66y3tuzn4wp3h02t7pZ:~# rm dump.rdb -rf root@iZuf66y3tuzn4wp3h02t7pZ:~# redis-cli 127.0.0.1:6379> set p1 1 OK 127.0.0.1:6379> set p2 1 OK 127.0.0.1:6379> set p3 3 OK root@iZuf66y3tuzn4wp3h02t7pZ:~# ls dump.rdb

It is also normal

Persistent trigger mechanism

• If the rules of save are satisfied, persistence will be triggered. For example, the above 60s operation of redis 3 times will trigger 1 persistence.
• When the flushall command is executed, persistence is also triggered, and the dump.db file is generated
• When exiting redis, persistence will also be triggered, and a dump.db file will be generated

The backup will automatically generate a dump.db file

How to restore persistent files

1. Just put the dump.db file in the startup directory of redis. When redis starts, the file will be read into the memory for data recovery.

2. You can do this by viewing the startup directory of redis

 127.0.0.1:6379> config get dir 1) "dir" 2) "/root"

The configuration of this piece, we basically don't need to modify too much, the default configuration is basically enough

Advantages of RDBs

• Suitable for large-scale data recovery
• Does not require high data integrity

Disadvantages of RDBs

• It needs to be operated at a certain time interval. If redis crashes unexpectedly, the last written data will be lost
• Fork child process needs to occupy a certain amount of space

AOF persistence method

What is AOF?

Record all our write commands, and execute all the records in the file when restoring

AOF is another persistence method of redis. It records each write operation in the form of a log, and records all the write operations performed by redis. Only appending files is allowed, and files are not allowed to be rewritten.

When redis starts, it will read the aof file to rebuild the database, that is to say, when redis restarts, it will execute the write instructions in the order of writing according to the content of the log file to complete data recovery.

aof saves the appendonly.aof file

 # Please check https://redis.io/topics/persistence for more information. appendonly no # The name of the append only file (default: "appendonly.aof") appendfilename "appendonly.aof" # appendfsync always appendfsync everysec # appendfsync no no-appendfsync-on-rewrite no auto-aof-rewrite-percentage 100 auto-aof-rewrite-min-size 64mb

auto-aof-rewrite-min-size 64mb

When the aof file is larger than 64 mb, a subprocess will be created to write a new aof file

Regarding the configuration of aof, basically everything else is to use the default configuration, we only need to open the aof mode

 appendonly yes

By default appendonly is not enabled. After we modify the configuration, restarting redis-server will take effect immediately.

Rewrite Rule Description

By default, aof appends infinitely to the file, and the file will inevitably become larger and larger.

After modifying redis.conf to aof mode, restart redis-server to see appendonly.aof file

The redis client connects to the server to operate redis, and simply set several values

 root@iZuf66y3tuzn4wp3h02t7pZ:/# redis-cli 127.0.0.1:6379> ping PONG 127.0.0.1:6379> set name xiaozhu OK 127.0.0.1:6379> set age 19 OK 127.0.0.1:6379> set hobby play OK 127.0.0.1:6379> set k1 v1 OK 127.0.0.1:6379> set k2 v2 OK 127.0.0.1:6379> set k3 v3 OK 127.0.0.1:6379> shutdown not connected>

View appendonly.aof file

The appendonly.aof file stores the records of the write commands we operate on redis

At this time, we think to modify some values in the appendonly.aof file

 set $2 k2 $2 v2 *3 $3 set $2 k3 $2 ashdkklasdjkkv3 # 修改了这一行

Start redis-server to see if the data can be recovered

 root@iZuf66y3tuzn4wp3h02t7pZ:/# redis-server /usr/local/redis/redis-6.2.5/redis.conf root@iZuf66y3tuzn4wp3h02t7pZ:/# redis-cli Could not connect to Redis at 127.0.0.1:6379: Connection refused not connected> root@iZuf66y3tuzn4wp3h02t7pZ:/# ps axu |grep redis root 1251 0.0 0.0 14436 1048 pts/0 S+ 14:55 0:00 grep --color=auto redis root@iZuf66y3tuzn4wp3h02t7pZ:/#

It was found that the redis-server failed to start because our appendonly.aof file has been artificially modified. At this time, we need to repair the file. Redis provides tools to modify the aof file, redis-check-aof

Use the command:

redis-check-aof --fix appendonly.aof

 # redis-check-aof --fix appendonly.aof 0x ce: Expected \r\n, got: 6864 AOF analyzed: size=223, ok_up_to=181, ok_up_to_line=47, diff=42 This will shrink the AOF from 223 bytes, with 42 bytes, to 181 bytes Continue? [y/N]: y Successfully truncated AOF root@iZuf66y3tuzn4wp3h02t7pZ:/# redis-server /usr/local/redis/redis-6.2.5/redis.conf root@iZuf66y3tuzn4wp3h02t7pZ:/# redis-cli 127.0.0.1:6379> ping PONG 127.0.0.1:6379> get k1 "v1"

After repairing the aof file, we start the redis-server again to restore the disk data, the recovery is successful, nice

Advantages and disadvantages of aof

Advantage

• Each operation reids will be recorded, the integrity of the file is good
• Sync every second, possibly losing a second of data
• Never sync, this efficiency is the highest

disadvantage

• Compared with the data file, the aof file will be much larger than the rdb file, and the repair speed is slower than that of the rdb file
• Aof runs slower than rdb, so the default configuration of redis is rdb persistence

summary

Brief description of the two persistence methods

• The RDB persistence method enables snapshot storage of data within a specified time interval
• The AOF persistence method records each write operation to the server. When the server restarts or goes down, the write operation command in these records will be re-executed to restore the data. The AOF command appends and saves each write operation to the file using the redis protocol. At the end, redis can also rewrite the aof file in the background. Yes, the size of the AOF file will not be too large.
• If the requirement is to only cache and only expect the data to exist when the server is running, then there is no need to do persistence

On and off of the two persistence methods

• Two persistence methods can be enabled at the same time

  • In this case, the redis restart will load the aof file first to restore the data, because usually the data set saved by the aof file is more complete than the data set of rdb
  • The rdb data set is not real-time. When using the two methods at the same time, the server will only find the aof file after restarting, so should we only use the aof file?

  The author of redis recommends not to use only aof files, because rdb is more suitable for backing up databases, because aof is constantly changing, it is not easy to back up, and when restarting quickly, rdb will not have potential bugs of aof, so keep rdb for it A bottom-up mechanism

performance advice

For rdb persistence

• Since the rdb file is only used for backing up data, it is recommended to only persist the rdb file on the slave, and it is enough to persist it once in 15 minutes, that is, this command save 900 1

• If you turn on aof persistence, the advantage is that in extreme cases, the data will not be lost for more than 2s. The startup script can simply load its own aof file, which is also costly

  • One of the costs is that doing so brings continuous IO operations
  • The second cost is that at the end of AOF rewriting, the blockage caused by writing new data to new files during the rewriting process is inevitable. As long as the hard disk permits, the frequency of aof rewriting should be reduced as much as possible.

for aof persistence

• The basic size value of aof rewrite is 64 mb, we can set it to more than 5g, and the default size exceeds 100% of the original size. Size rewrite, this parameter can be set to a reasonable parameter

• If the aof mode is not turned on, it is possible to achieve high availability only by master-slave replication, which can save a lot of IO consumption and reduce the performance fluctuation of the system caused by rewriting. This is still costly.

  • One of the costs is that if the primary and secondary redis hang up at the same time (such as power failure), the data will be lost for more than ten minutes. The startup script also needs to compare the primary and secondary rdb files and load the newer rdb file.

References:

redis_doc

Welcome to like, follow, favorite

Friends, your support and encouragement are the motivation for me to persist in sharing and improve quality

Okay, here it is this time

Technology is open, and our mentality should be open. Embrace change, live in the sun, and move forward.

I'm the little devil boy Nezha , welcome to like, follow and collect, see you next time~