[Redis series] Redis learning eleven, detailed explanation and actual combat of redis sentinel mode

The master-slave replication we talked about last time is built like this

The host can read and write

The slave can only read, not write

Come to think of it, can we do the same?

Multiple redis-servers are connected end to end

Then when we deploy it is 6379 – 6380 – 6381

At this point, if the host 6379 goes down, will the 6380 become the host?

 127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=6380,state=online,offset=0,lag=1
master_failover_state:no-failover
master_replid:f1e3db9e5e438f5d98e4cad23f684b12d790ae56
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:0

We can see that the 6379 has a slave, and the 6380 itself is a slave (although the 6380 is the master of the 6381)

At this time, the 6379 was shut down, and it was found that the 6380 was still a slave

 127.0.0.1:6379> shudown
not connected>

We artificially changed the server of reids 6380 to the host to see if we can grab the master back after the 6379 redis-server is up.

Use slaveof no one to set yourself as master

Start 6379 redis-server

 redis-server /usr/local/redis/redis-6.2.5/6379.conf

Found that 6380 is still the host, 6379 has become the bare commander

In actual projects, we will definitely not adopt the deployment methods mentioned above and in the previous article, their ability to resist risks is too low

Because in the actual production environment, the host machine is down. If the slave machine cannot become the host machine, can it not be possible to do write operations until the host machine responds? this is a serious problem

Let's take a closer look at how the sentinel mode solves the above problems

Sentinel Mode

The mode of automatically electing the master

introduce

The way to actively switch the master is:

When the host server is down, in the past, we needed to manually change a certain slave server to the host server, which required manual processing, time-consuming and labor-intensive, and would cause the service to be unavailable for a period of time. This approach is impossible. taken

So with the sentinel mode, the sentinel mode is the sentinel architecture provided by the redis 2.8 version to solve the above problems

Sentinel mode , which can monitor whether the host server in the background is faulty. If there is a fault, a slave server will be voted to be the host.

Sentinel mode is a special mode, Redis provides sentinel commands

Sentinel is actually an independent process , as a process, it will run independently

The principle is that the sentinel monitors multiple running Redis instances by sending commands and waiting for the Redis server to respond.

Practical drill

In the architecture diagram above, the sentinel has two functions:

• By sending commands, the Redis server returns monitoring status information, including
• If the sentinel detects that the master server is down, it will automatically switch the slave to the master, and then notify other slave servers through publish and subscribe, modify the configuration file, and make him the master

However, if a sentinel monitors a redis cluster, the possibility of problems will be much greater. Therefore, our sentinels can also be clustered, and each sentinel will monitor each other, just like the picture below

Subjective offline

For example, let's take an example. If the master server is down, then one of the sentinels will detect that the system will not immediately execute the failover process. It is only the current sentinel that judges that the master is unavailable. This is subjective offline .

objectively offline

When the other two sentinels also find that the master server is unavailable, a vote will be generated between the sentinels. We will write the specific voting algorithm later. The voting structure is initiated by a sentinel to perform failover failover operations. After the switch is successful , it will use the publish-subscribe mode to let each monitoring sentinel switch the server it monitors to the master server. This is the objective offline

Let's configure and start a sentinel:

Also in our configuration file directory, which is the same level as redis, create a sentinel.conf file and write the configuration. This file will also generate a redis by default.

As above, the command we are concerned with is

 sentinel monitor mymaster   127.0.0.1   6379   1

Configure a sentinel to monitor the redis cluster

Start the sentry process

 root@iZuf66y3tuzn4wp3h02t7pZ:/usr/local/redis/redis-6.2.5# redis-sentinel sentinel.conf
18148:X 26 Aug 2021 22:22:36.187 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
18148:X 26 Aug 2021 22:22:36.187 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=18148, just started
18148:X 26 Aug 2021 22:22:36.187 # Configuration loaded
18148:X 26 Aug 2021 22:22:36.188 * monotonic clock: POSIX clock_gettime
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 6.2.5 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 26379
 |    `-._   `._    /     _.-'    |     PID: 18148
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           https://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

18148:X 26 Aug 2021 22:22:36.189 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
18148:X 26 Aug 2021 22:22:36.193 # Sentinel ID is 7e01f5aa31aadb7fc54ed8ef2579c77120682dc9
18148:X 26 Aug 2021 22:22:36.193 # +monitor master mymaster 127.0.0.1 6379 quorum 1
18148:X 26 Aug 2021 22:22:36.193 * +slave slave 127.0.0.1:6380 127.0.0.1 6380 @ mymaster 127.0.0.1 6379
18148:X 26 Aug 2021 22:22:36.196 * +slave slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6379

We can see that after the sentinel process is started, it starts to monitor the redis cluster and outputs the ip and port of the monitored redis

At this point, we check the information of a master redis server, we can see that there are currently 2 slaves, and now we let the host down

Looking at the log of the sentinel, we can see that the sentinel process starts to vote for the host after 30 seconds

According to the log, we can see that after the host of 6379 went down, the sentinel elected the slave of 6381 as the new host, and the automatic failure recovery was successful, nice

Check the master host process of 6381 and see that it is the master and has 1 slave

Since the default configuration of 6379 is the host, when the 6379 server is started again, the 6379 becomes a polished commander, and the slave becomes 0.

 root@iZuf66y3tuzn4wp3h02t7pZ:~# redis-server /usr/local/redis/redis-6.2.5/6379.conf
root@iZuf66y3tuzn4wp3h02t7pZ:~# redis-cli -p 6379
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:80843f8a6497705983f6463b92d71ebd451ef385
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

There are not many configuration items in the sentinel.conf configuration file. Let's explain them one by one in detail:

The detailed configuration of sentry mode is as follows:

• port

Sentinel instance running port, the default is 26379 , if there is a sentinel cluster, we also need to configure each sentinel port

• dir

Sentinel's working directory

• sentinel monitor <master-name> <ip> <redis-port> <quorum>

The ip port of the redis master node monitored by sentinel

master-name , the master node name that can be named by itself can only be composed of letters AZ, numbers 0-9, and these three characters " . - _ ".

How many sentinels are configured in quorum , and the sentinels uniformly think that the master node is out of contact, then objectively think that the master node is out of contact

• sentine1 auth-pass <master-name> <password>

When the requirepass foobared authorization password is enabled in the Redis instance, all clients connecting to the redis instance must provide the password

Set the sentinel password to connect the master and slave. Note that the same verification password must be set for the master and slave

• sentinel down-after-mi 11i seconds <master-name> <mi 11iseconds>

After the specified number of milliseconds, the master node does not respond to the sentinel sentine1. At this time, the sentinel subjectively believes that the master node is offline by default for 30 seconds.

• sentinel paralle1-syncs <master-name> <numslaves>

Specifies how many slaves can synchronize the new master at the same time when failover master-slave switchover occurs

The lower the number, the longer it will take to complete the failover

But if this number is larger, it means that more slaves are unavailable due to replication

You can set this value to 1 to ensure that only one slave is in a state that cannot process command requests at a time

• sentinel failover-timeout <master-name> <milliseconds>

The failover-timeout can be used in the following ways:

1. The interval between two failovers of the same sentinel to the same master

2. When a slave synchronizes data from a wrong master, the time is calculated

Until the slave is corrected to sync data to the correct master

3. The time it takes to cancel an ongoing failover

4. When performing failover , configure the maximum time required for all slaves to point to the new master

However, even after this timeout, the slaves will still be correctly configured to point to the master, but not according to the rules configured by parallel-syncs

5. The default time is three minutes

• sentinel notification-script <master-name> <script-path>

When sentinel has any warning level event (such as subjective failure and objective failure of redis instance, etc.), this script will be called

At this time, the script should notify the system administrator about the abnormal operation of the system by email, SMS, etc.

When calling the script, two parameters will be passed to the script

1. The type of event

2. Description of the event

If the script path is configured in the sentinel . conf configuration file, then it must be ensured that the script exists in this path and is executable, otherwise sentinel cannot start successfully.

• sentinel client-reconfig-script <master-name> <script-path>

This configuration is the client-side reconfigure master node parameter script

When a master changes due to failover, this script will be called to notify relevant clients that the master address has changed

The following parameters will be passed to the script when it is called:

<master-name>

<role> , is one of Teader or Observer - one

<state> , usually failover

<from-ip>

<from-port>

<to-ip> <to-port>

The parameters from-ip, from-port, to-ip, to-port are used to communicate with the old master and the new master (ie the old slave)

 port 26379

dir /tmp

sentinel monitor mymaster   127.0.0.1   6379   2

sentine1 auth-pass mymaster MySUPER--secret-0123passwOrd

sentine1 down-after-mi 11iseconds mymaster 30000

sentine1 paralle1-syncs mymaster 1

sentine1 fai lover-ti meout mymaster 180000

sentine1 notificati on-script mymaster /var/redis/notify. sh

sentine1 client-reconfig-script mymaster /var/redis/reconfig.sh

References:

redis_doc

Welcome to like, follow, favorite

Friends, your support and encouragement are the motivation for me to persist in sharing and improve quality

Okay, here it is this time

Technology is open, and our mentality should be open. Embrace change, live in the sun, and move forward.

I'm the little devil boy Nezha , welcome to like, follow and collect, see you next time~