redis哨兵集群

Sentinel(哨兵)进程是用于监控redis集群中Master主服务器工作的状态,在Master主服务器发生故障的时候,可以实现MasterSlave服务器的切换,保证系统的高可用,其已经被集成在redis2.6+的版本中,Redis的哨兵模式到了2.8版本之后就稳定了下来。一般在生产环境也建议使用Redis的2.8版本的以后版本。

哨兵(Sentinel) 是一个分布式系统,你可以在一个架构中运行多个哨兵(sentinel) 进程,这些进程使用流言协议(gossipprotocols)来接收关于Master主服务器是否下线的信息,并使用投票协议(Agreement Protocols)来决定是否执行自动故障迁移,以及选择哪个Slave作为新的Master

每个哨兵(Sentinel)进程会向其它哨兵(Sentinel)、MasterSlave定时发送消息,以确认对方是否”活”着,如果发现对方在指定配置时间(可配置的)内未得到回应,则暂时认为对方已掉线,也就是所谓的”主观认为宕机” ,英文名称:Subjective Down,简称SDOWN。有主观宕机,肯定就有客观宕机。

当“哨兵群”中的多数Sentinel进程在对Master主服务器做出 SDOWN 的判断,并且通过 SENTINEL is-master-down-by-addr 命令互相交流之后,得出的Master Server下线判断,这种方式就是“客观宕机”,英文名称是:Objectively Down, 简称 ODOWN。通过一定的vote算法,从剩下的slave从服务器节点中,选一台提升为Master服务器节点,然后自动修改相关配置,并开启故障转移(failover)。

Redis-Sentinel

Redis-Sentinelredis官方推荐的高可用性解决方案,当用redismaster-slave的高可用时,如果master本身宕机,redis本身或者客户端都没有实现主从切换的功能。

redis-sentinel就是一个独立运行的进程,用于监控多个master-slave集群,自动发现master宕机,进行自动切换slave > master

sentinel主要功能如下:

  • 不时的监控redis是否良好运行,如果节点不可达就会对节点进行下线标识
  • 如果被标识的是主节点,sentinel就会和其他的sentinel节点“协商”,如果其他节点也人为主节点不可达,就会选举一个sentinel节点来完成自动故障转义
  • master-slave进行切换后,master_redis.confslave_redis.confsentinel.conf的内容都会发生改变,即master_redis.conf中会多一行slaveof的配置,sentinel.conf的监控目标会随之调换

redis主从复制背景问题

Redis主从复制可将主节点数据同步给从节点,从节点此时有两个作用:

  • 一旦主节点宕机,从节点作为主节点的备份可以随时顶上来。
  • 扩展主节点的读能力,分担主节点读压力。

但是问题是:

  • 一旦主节点宕机,从节点上位,那么需要人为修改所有应用方的主节点地址(改为新的master地址),还需要命令所有从节点复制新的主节点

那么这个问题,redis-sentinel就可以解决了

主从复制架构

Redis Sentinel架构

sentinelredis的一个进程,但是不存储数据,只用来监控redis

sentinel会通过命令连接向被监视的主从服务器发送“hello”信息,该消息包含sentinelip、端口号、id等内容,以此来向其他sentinel宣告自己存在。与此同时sentinel会通过订阅连接接收其他sentinel的“hello”信息,以此来发现监视同一个主服务器的其他sentinel

redis哨兵相关命令

1
2
3
redis-cli info #查看redis数据库信息
redis-cli info replication #查看redis的复制授权信息
redis-cli info sentinel #查看redis的哨兵信息

示例

环境准备

1
2
3
127.0.0.1 6379 主库master
127.0.0.1 6380 从库slave
127.0.0.1 6381 从库slave

配置文件

6379配置文件

1
2
3
4
5
6
7
8
root@Tony-PC:/tmp/redis-config/6379# cat 6379.conf
daemonize yes
port 6379
pidfile /tmp/redis-config/6379/6379.pid
loglevel notice
logfile /tmp/redis-config/6379/6379.log
dir /tmp/redis-config
dbfilename 6379.rdb

6380配置文件:

使用sed命令,将6379.conf中的的6379替换为6380,替换后的结果写入到6380.conf

1
root@Tony-PC:/tmp/redis-config/6379# sed "s/6379/6380/g" redis-6379.conf > ../6380/6380.conf
1
2
3
4
5
6
7
8
9
root@Tony-PC:/tmp/redis-config/6379# cat ../6380/6380.conf
daemonize yes
port 6380
pidfile /tmp/redis-config/6380/6380.pid
loglevel notice
logfile /tmp/redis-config/6380/6380.log
dir /tmp/redis-config
dbfilename 6380.rdb
slaveof 127.0.0.1 6379 # 指明主的地址

6381配置文件:

使用sed命令,将6379.conf中的的6379替换为6381,替换后的结果写入到6381.conf

1
root@Tony-PC:/tmp/redis-config/6379# sed "s/6379/6381/g" redis-6379.conf > ../6381/6381.conf
1
2
3
4
5
6
7
8
9
root@Tony-PC:/tmp/redis-config/6379# cat ../6381/6381.conf
daemonize yes
port 6381
pidfile /tmp/redis-config/6381/6381.pid
loglevel notice
logfile /tmp/redis-config/6381/6381.log
dir /tmp/redis-config
dbfilename 6381.rdb
slaveof 127.0.0.1 6379 # 指明主的地址

启动redis3个服务

1
2
3
4
5
6
7
8
9
10
11
12
13
root@Tony-PC:/tmp/redis-config/6379# redis-server 6379.conf
root@Tony-PC:/tmp/redis-config/6379# redis-server ../6380/6380.conf
root@Tony-PC:/tmp/redis-config/6379# redis-server ../6381/6381.conf

root@Tony-PC:/tmp/redis-config/6379# ps -ef | grep redis
root 5570 5546 0 10:54 pts/3 00:00:00 vim /etc/redis/redis.conf
tony 5782 1 0 10:55 ? 00:00:00 /bin/bash /usr/bin/dde-file-manager-pkexec file:///var/log/redis
root 5784 5782 0 10:55 ? 00:00:43 /usr/bin/dde-file-manager file:///var/log/redis -w /home/tony
root 14081 7206 0 14:45 pts/4 00:00:00 redis-cli
root 16443 1 0 15:48 ? 00:00:00 redis-server *:6379
root 16448 1 0 15:48 ? 00:00:00 redis-server *:6380
root 16454 1 0 15:49 ? 00:00:00 redis-server *:6381
root 16464 7362 0 15:49 pts/5 00:00:00 grep redis

查看主从状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
root@Tony-PC:~# redis-cli
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:2
slave0:ip=127.0.0.1,port=6380,state=online,offset=252,lag=1
slave1:ip=127.0.0.1,port=6381,state=online,offset=252,lag=1
master_replid:b8566352985ea4d4dd23a1d2a6f333f02c1d5c0c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:252
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:252


root@Tony-PC:/tmp/redis-config# redis-cli -p 6380
127.0.0.1:6380> info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:6
master_sync_in_progress:0
slave_repl_offset:266
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:b8566352985ea4d4dd23a1d2a6f333f02c1d5c0c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:266
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:266


root@Tony-PC:/tmp/redis-config# redis-cli -p 6381
127.0.0.1:6381> info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:3
master_sync_in_progress:0
slave_repl_offset:280
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:b8566352985ea4d4dd23a1d2a6f333f02c1d5c0c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:280
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:15
repl_backlog_histlen:266

配置Sentinel

sentinel-26379.conf文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
// Sentinel节点的端口
port 26379
daemonize yes
dir /tmp/redis-config/6379/
logfile "26379.log"

// 当前Sentinel节点监控 127.0.0.1:6379 这个主节点
// 2代表判断主节点失败至少需要2个Sentinel节点节点同意
// mymaster是主节点的别名
sentinel monitor mymaster 127.0.0.1 6379 2

//每个Sentinel节点都要定期PING命令来判断Redis数据节点和其余Sentinel节点是否可达,如果超过30000毫秒30s且没有回复,则判定不可达
sentinel down-after-milliseconds mymaster 30000

//当Sentinel节点集合对主节点故障判定达成一致时,Sentinel领导者节点会做故障转移操作,选出新的主节点,
原来的从节点会向新的主节点发起复制操作,限制每次向新的主节点发起复制操作的从节点个数为1
sentinel parallel-syncs mymaster 1

//故障转移超时时间为180000毫秒
sentinel failover-timeout mymaster 180000

sentinel-26380.conf文件

1
2
3
4
5
6
7
8
port 26380
daemonize yes
dir "/tmp/redis-config/6380"
logfile "26380.log"
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000

sentinel-26381.conf文件

1
2
3
4
5
6
7
8
port 26381
daemonize yes
dir "/tmp/redis-config/6381"
logfile "26381.log"
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 30000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 180000

启动哨兵

1
2
3
root@Tony-PC:/tmp/redis-config/6379# redis-sentinel sentinel-26379.conf
root@Tony-PC:/tmp/redis-config/6379# redis-sentinel ../6380/sentinel-26380.conf
root@Tony-PC:/tmp/redis-config/6379# redis-sentinel ../6381/sentinel-26381.conf

查看哨兵状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
root@Tony-PC:~# redis-cli -p 26379
127.0.0.1:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=127.0.0.1:6379
,slaves=2,sentinels=3


root@Tony-PC:/tmp/redis-config# redis-cli -p 26380
127.0.0.1:26380> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=127.0.0.1:6379,slaves=2,senti
nels=3


root@Tony-PC:/tmp/redis-config# redis-cli -p 26381
127.0.0.1:26381> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=127.0.0.1:6379,slaves=2,senti
nels=3

看到最后一条信息正确即成功了哨兵,哨兵主节点名字叫做mymaster,状态ok,监控地址是127.0.0.1:6379,有两个从节点,3个哨兵

杀掉主库

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
root@Tony-PC:/tmp/redis-config/6379# ps -ef | grep redis
root 5570 5546 0 10:54 pts/3 00:00:00 vim /etc/redis/redis.conf
tony 5782 1 0 10:55 ? 00:00:00 /bin/bash /usr/bin/dde-file-manager-pkexec file:///var/log/redis
root 5784 5782 0 10:55 ? 00:01:05 /usr/bin/dde-file-manager file:///var/log/redis -w /home/tony
root 17517 1 0 16:14 ? 00:00:10 /opt/sublime_text_3/sublime_text --class=sublime-text-dev /tmp/redis
-config/6381/6381.confroot 20047 1 0 17:21 ? 00:00:07 /usr/share/code/code --unity-launch /tmp/redis-config/6381/sentinel-
26381.confredis 22074 1 0 17:35 ? 00:00:00 /usr/bin/redis-sentinel *:26379 [sentinel]
root 22515 1 0 17:36 ? 00:00:00 redis-server *:6379
root 22522 1 0 17:36 ? 00:00:00 redis-server *:6380
root 22528 1 0 17:36 ? 00:00:00 redis-server *:6381
root 22610 1 0 17:38 ? 00:00:00 redis-sentinel *:26380 [sentinel]
root 22615 1 0 17:38 ? 00:00:00 redis-sentinel *:26381 [sentinel]
root 22897 5000 0 17:43 pts/1 00:00:00 redis-cli
root 22898 7206 0 17:43 pts/4 00:00:00 redis-cli -p 6380
root 22899 16613 0 17:44 pts/6 00:00:00 redis-cli -p 6381
root 22971 7362 0 17:45 pts/5 00:00:00 grep redis


root@Tony-PC:/tmp/redis-config/6379# kill -9 22515


root@Tony-PC:/tmp/redis-config/6379# ps -ef | grep redis
root 5570 5546 0 10:54 pts/3 00:00:00 vim /etc/redis/redis.conf
tony 5782 1 0 10:55 ? 00:00:00 /bin/bash /usr/bin/dde-file-manager-pkexec file:///var/log/redis
root 5784 5782 0 10:55 ? 00:01:05 /usr/bin/dde-file-manager file:///var/log/redis -w /home/tony
root 17517 1 0 16:14 ? 00:00:10 /opt/sublime_text_3/sublime_text --class=sublime-text-dev /tmp/redis
-config/6381/6381.confroot 20047 1 0 17:21 ? 00:00:07 /usr/share/code/code --unity-launch /tmp/redis-config/6381/sentinel-
26381.confredis 22074 1 0 17:35 ? 00:00:00 /usr/bin/redis-sentinel *:26379 [sentinel]
root 22522 1 0 17:36 ? 00:00:00 redis-server *:6380
root 22528 1 0 17:36 ? 00:00:00 redis-server *:6381
root 22610 1 0 17:38 ? 00:00:00 redis-sentinel *:26380 [sentinel]
root 22615 1 0 17:38 ? 00:00:00 redis-sentinel *:26381 [sentinel]
root 22897 5000 0 17:43 pts/1 00:00:00 redis-cli
root 22898 7206 0 17:43 pts/4 00:00:00 redis-cli -p 6380
root 22899 16613 0 17:44 pts/6 00:00:00 redis-cli -p 6381
root 22981 7362 0 17:45 pts/5 00:00:00 grep redis

查看从库状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
127.0.0.1:6380> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=6381,state=online,offset=92232,lag=0
master_replid:4378f66f92d63d4bebaccd054f9fdc2fb874e9a7
master_replid2:313b7f50e569c4d17cacaefbf869b105bd6a92c0
master_repl_offset:92498
second_repl_offset:90348
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:92498


127.0.0.1:6381> info replication
# Replication
role:slave
master_host:127.0.0.1
master_port:6380
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:95319
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:4378f66f92d63d4bebaccd054f9fdc2fb874e9a7
master_replid2:313b7f50e569c4d17cacaefbf869b105bd6a92c0
master_repl_offset:95319
second_repl_offset:90348
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:437
repl_backlog_histlen:94883

以上可以看到主库已有6379变为了6380,证明哨兵设置成功

注:本人使用apt-get安装的redis,发现没有把redis-sentinel上去,使用apt-get install redis-sentinel 单独安装下就可以了