搭建 Redis 集群(手动、利用 redis-trib)

Redis Cluster 是 Redis 提供的分布是解决方案,有效的解决了 Redis 分布式方面的需求,当遇到单机内存、并发、流量等瓶颈时,就可以考虑 cluster 架构达到负载均衡的目的。

Redis Cluster 采用虚拟分槽(slot),虚拟槽分区巧妙地利用了 hash 空间,使用分散度良好的哈希函数将所有数据映射到一个固定范围内的整数集合,整数定义为槽(slot)。Redis Cluster 槽的范围是 0 ~ 16383,槽是集群内数据管理和迁移的基本单位。采用大范围槽的主要目的是为了方便数据的拆分和集群的扩展,每个节点负责一定数量的槽。

节点准备

不管是手动搭建还是利用 redis-trib,首先我们需要一定数量的 redis 服务器,本篇中我们采用本机启动 6 个节点,3 主 3 从。

创建配置文件

1
2
3
4
5
port 8001
cluster-enabled yes
cluster-config-file node-8001.conf
cluster-node-timeout 5000
appendonly yes

对于每个不同的节点,修改上述配置文件中 8001 即可。我们创建了 800180028003800480058006 六个节点。

Redis Cluster 启动时会同时占用 当前端口号+10000 的第二个高端口号,该端口号的作用为:

This second high port is used for the Cluster bus, that is a node-to-node communication channel using a binary protocol

启动 Redis

我们使用 docker 来启动 redis:

1
2
3
4
5
6
redis-server 8001/redis-8001.conf 2>&1 > 8001/logs &
redis-server 8002/redis-8002.conf 2>&1 > 8002/logs &
redis-server 8003/redis-8003.conf 2>&1 > 8003/logs &
redis-server 8004/redis-8004.conf 2>&1 > 8004/logs &
redis-server 8005/redis-8005.conf 2>&1 > 8005/logs &
redis-server 8006/redis-8006.conf 2>&1 > 8006/logs &

查看端口情况

1
2
3
4
5
6
7
8
9
10
11
12
13
$ netstat -tulnp
tcp 0 0 0.0.0.0:18001 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:18002 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:18003 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:18004 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:18005 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:18006 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8001 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8002 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8003 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8004 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8005 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8006 0.0.0.0:* LISTEN -

手动搭建

节点握手

使用 redis-cli 连接 redis cluster 的时候需要带上 -c 选项,它提供了基础的 redis cluster 支持。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
$ redis-cli -c -p 8001
127.0.0.1:8001> cluster nodes # 查看集群节点状态
b1e03eeee490f23507bb31fd1682b8d31b7a48a2 :8001@18001 myself,master - 0 0 0 connected
127.0.0.1:8001> cluster meet 127.0.0.1 8002 # 和 8002 握手
OK
127.0.0.1:8001> cluster meet 127.0.0.1 8003 # 和 8003 握手
OK
127.0.0.1:8001> cluster meet 127.0.0.1 8004 # 和 8004 握手
OK
127.0.0.1:8001> cluster meet 127.0.0.1 8005 # 和 8005 握手
OK
127.0.0.1:8001> cluster meet 127.0.0.1 8006 # 和 8006 握手
OK
127.0.0.1:8001> cluster nodes # 查看集群节点状态
669683f96c894eab66ace526257a9a062f2d7c44 127.0.0.1:8003@18003 master - 0 1533509076000 2 connected
9bd7f3db4ee921ce697ecd7f2a9a583c0056c6f7 127.0.0.1:8006@18006 master - 0 1533509075160 0 connected
0cfac6037e73143019eefe30cf00a54093bb2940 127.0.0.1:8004@18004 master - 0 1533509076000 3 connected
991b20166c43bceff023354d6c06626fcf552c7c 127.0.0.1:8005@18005 master - 0 1533509076572 3 connected
b1e03eeee490f23507bb31fd1682b8d31b7a48a2 127.0.0.1:8001@18001 myself,master - 0 1533509074000 0 connected
72b006c5a1faee83e51b2972a0d602e7368f03cb 127.0.0.1:8002@18002 master - 0 1533509077000 1 connected
127.0.0.1:8001> cluster info # 查看集群信息
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:0
cluster_current_epoch:5
cluster_my_epoch:0
cluster_stats_messages_ping_sent:170
cluster_stats_messages_pong_sent:182
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:357
cluster_stats_messages_ping_received:182
cluster_stats_messages_pong_received:175
cluster_stats_messages_received:357

分配槽

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# 给 master-1 分配槽
$ redis-cli -p 8001 cluster addslots {0..5461}
OK
# 给 master-2 分配槽
$ redis-cli -p 8002 cluster addslots {5462..10922}
OK
# 给 master-3 分配槽
$ redis-cli -p 8003 cluster addslots {10923..16383}
OK
# 查看分配槽之后的集群信息
$ redis-cli -c -p 8001
127.0.0.1:8001> cluster nodes
669683f96c894eab66ace526257a9a062f2d7c44 127.0.0.1:8003@18003 master - 0 1533509385398 2 connected 10923-16383
9bd7f3db4ee921ce697ecd7f2a9a583c0056c6f7 127.0.0.1:8006@18006 master - 0 1533509385000 4 connected
0cfac6037e73143019eefe30cf00a54093bb2940 127.0.0.1:8004@18004 master - 0 1533509384391 5 connected
991b20166c43bceff023354d6c06626fcf552c7c 127.0.0.1:8005@18005 master - 0 1533509385598 3 connected
b1e03eeee490f23507bb31fd1682b8d31b7a48a2 127.0.0.1:8001@18001 myself,master - 0 1533509384000 0 connected 0-5461
72b006c5a1faee83e51b2972a0d602e7368f03cb 127.0.0.1:8002@18002 master - 0 1533509383586 1 connected 5462-10922
127.0.0.1:8001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:5
cluster_my_epoch:0
cluster_stats_messages_ping_sent:647
cluster_stats_messages_pong_sent:680
cluster_stats_messages_meet_sent:5
cluster_stats_messages_sent:1332
cluster_stats_messages_ping_received:680
cluster_stats_messages_pong_received:652
cluster_stats_messages_received:1332

配置从节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ redis-cli -c -p 8004
127.0.0.1:8004> cluster nodes
9bd7f3db4ee921ce697ecd7f2a9a583c0056c6f7 127.0.0.1:8006@18006 master - 0 1533509619540 4 connected
0cfac6037e73143019eefe30cf00a54093bb2940 127.0.0.1:8004@18004 myself,master - 0 1533509618000 5 connected
72b006c5a1faee83e51b2972a0d602e7368f03cb 127.0.0.1:8002@18002 master - 0 1533509619540 1 connected 5462-10922
b1e03eeee490f23507bb31fd1682b8d31b7a48a2 127.0.0.1:8001@18001 master - 0 1533509619000 0 connected 0-5461
669683f96c894eab66ace526257a9a062f2d7c44 127.0.0.1:8003@18003 master - 0 1533509619000 2 connected 10923-16383
991b20166c43bceff023354d6c06626fcf552c7c 127.0.0.1:8005@18005 master - 0 1533509620546 3 connected
127.0.0.1:8004> cluster replicate b1e03eeee490f23507bb31fd1682b8d31b7a48a2
OK
127.0.0.1:8004> connect 127.0.0.1 8005
127.0.0.1:8005> cluster replicate 72b006c5a1faee83e51b2972a0d602e7368f03cb
OK
127.0.0.1:8005> connect 127.0.0.1 8006
127.0.0.1:8006> cluster replicate 669683f96c894eab66ace526257a9a062f2d7c44
OK

查看集群最终节点状态

1
2
3
4
5
6
7
8
$ redis-cli -c -p 8001
127.0.0.1:8001> cluster nodes
669683f96c894eab66ace526257a9a062f2d7c44 127.0.0.1:8003@18003 master - 0 1533509729381 2 connected 10923-16383
9bd7f3db4ee921ce697ecd7f2a9a583c0056c6f7 127.0.0.1:8006@18006 slave 669683f96c894eab66ace526257a9a062f2d7c44 0 1533509728877 4 connected
0cfac6037e73143019eefe30cf00a54093bb2940 127.0.0.1:8004@18004 slave b1e03eeee490f23507bb31fd1682b8d31b7a48a2 0 1533509730588 5 connected
991b20166c43bceff023354d6c06626fcf552c7c 127.0.0.1:8005@18005 slave 72b006c5a1faee83e51b2972a0d602e7368f03cb 0 1533509730388 3 connected
b1e03eeee490f23507bb31fd1682b8d31b7a48a2 127.0.0.1:8001@18001 myself,master - 0 1533509728000 0 connected 0-5461
72b006c5a1faee83e51b2972a0d602e7368f03cb 127.0.0.1:8002@18002 master - 0 1533509730000 1 connected 5462-10922

测试数据

使用 cluster keyslot 来查看 key 会被分配到哪个槽中

1
2
3
4
5
6
127.0.0.1:8001> cluster keyslot key:{10}:111
(integer) 247 # 247 槽,属于 master-1
127.0.0.1:8001> cluster keyslot key:{test}:111
(integer) 6918 # 6918 槽,属于 master-2
127.0.0.1:8001> cluster keyslot key:{hu}:111
(integer) 11441 # 11441 槽,属于 master-3

为上面三个 key 设置值:

1
2
3
4
5
6
7
8
127.0.0.1:8001> set key:{10}:111 value_of_key:{10}:111
OK
127.0.0.1:8001> set key:{test}:111 value_of_key:{test}:111
-> Redirected to slot [6918] located at 127.0.0.1:8002
OK
127.0.0.1:8002> set key:{hu}:111 value_of_key:{hu}:111
-> Redirected to slot [11441] located at 127.0.0.1:8003
OK

可以看到,在 master-1 上设置 key:{test}:111 时自动重定向到了 master-2,在 master-2 上设置 key:{hu}:111 时自动重定向到了 master-3。

获取值:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
127.0.0.1:8004> get key:{test}:111
-> Redirected to slot [6918] located at 127.0.0.1:8002
"value_of_key:{test}:111"
127.0.0.1:8002> get key:{10}:111
-> Redirected to slot [247] located at 127.0.0.1:8001
"value_of_key:{10}:111"
127.0.0.1:8001> connect 127.0.0.1 8005
127.0.0.1:8005> get key:{hu}:111
-> Redirected to slot [11441] located at 127.0.0.1:8003
"value_of_key:{hu}:111"
127.0.0.1:8003> connect 127.0.0.1 8006
127.0.0.1:8006> get key:{hu}:111
-> Redirected to slot [11441] located at 127.0.0.1:8003
"value_of_key:{hu}:111"

可以看到在获取值的时候,如果不是本节点所持有的槽,会自动重定向到相应的主节点去获取(即使当前节点是对应主节点的 slave,也就是说 slave 在 redis cluster 中只起到了替换主节点的作用)。

利用 redis-trib 搭建

Redis 在其源码包中附带了一个 ruby 脚本 redis-trib.rb 用来快速搭建 redis cluster,并且 Redis 官网的 tutorial 也指出:

In practical terms redis-trib here did very little to help us, it just sent a CLUSTER MEET message to the node, something that is also possible to accomplish manually. However redis-trib also checks the state of the cluster before to operate, so it is a good idea to perform cluster operations always via redis-trib even when you know how the internals work.

之前我们已经启动了 6 个节点,下面我们用 redis-trib 来搭建集群:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
./redis-trib.rb create --replicas 1 127.0.0.1:8001 127.0.0.1:8002 127.0.0.1:8003 127.0.0.1:8004 127.0.0.1:8005 127.0.0.1:8006
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
127.0.0.1:8001
127.0.0.1:8002
127.0.0.1:8003
Adding replica 127.0.0.1:8005 to 127.0.0.1:8001
Adding replica 127.0.0.1:8006 to 127.0.0.1:8002
Adding replica 127.0.0.1:8004 to 127.0.0.1:8003
>>> Trying to optimize slaves allocation for anti-affinity
[WARNING] Some slaves are in the same host as their master
M: e48d4589b0197a34863b0a3567098fa45ed62a88 127.0.0.1:8001
slots:0-5460 (5461 slots) master
M: c4374d6f822233c0c7ae899caf4c59a254965f36 127.0.0.1:8002
slots:5461-10922 (5462 slots) master
M: 5c4f74000b5923b21b7e62152ffd903ed6e299a7 127.0.0.1:8003
slots:10923-16383 (5461 slots) master
S: d1c3c556fc2ca29053a11fa27dd86e537bcb7ca8 127.0.0.1:8004
replicates 5c4f74000b5923b21b7e62152ffd903ed6e299a7
S: 88e2c0f6a29a1f15b5459ad4b8192aa81530c9f5 127.0.0.1:8005
replicates e48d4589b0197a34863b0a3567098fa45ed62a88
S: 69655d72f2dca0419855b28f1fe26600ff35bc86 127.0.0.1:8006
replicates c4374d6f822233c0c7ae899caf4c59a254965f36
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join....
>>> Performing Cluster Check (using node 127.0.0.1:8001)
M: e48d4589b0197a34863b0a3567098fa45ed62a88 127.0.0.1:8001
slots:0-5460 (5461 slots) master
1 additional replica(s)
S: 69655d72f2dca0419855b28f1fe26600ff35bc86 127.0.0.1:8006
slots: (0 slots) slave
replicates c4374d6f822233c0c7ae899caf4c59a254965f36
S: 88e2c0f6a29a1f15b5459ad4b8192aa81530c9f5 127.0.0.1:8005
slots: (0 slots) slave
replicates e48d4589b0197a34863b0a3567098fa45ed62a88
M: c4374d6f822233c0c7ae899caf4c59a254965f36 127.0.0.1:8002
slots:5461-10922 (5462 slots) master
1 additional replica(s)
S: d1c3c556fc2ca29053a11fa27dd86e537bcb7ca8 127.0.0.1:8004
slots: (0 slots) slave
replicates 5c4f74000b5923b21b7e62152ffd903ed6e299a7
M: 5c4f74000b5923b21b7e62152ffd903ed6e299a7 127.0.0.1:8003
slots:10923-16383 (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

这样就完成了,整个过程中我们只需要在 Can I set the above configuration? (type ‘yes’ to accept): 的地方输入 yes,redis-trib 会为我们完成手动需要的所有事情。检查集群状态:

1
2
3
4
5
6
7
8
$ redis-cli -c -p 8001
127.0.0.1:8001> cluster nodes
e48d4589b0197a34863b0a3567098fa45ed62a88 127.0.0.1:8001@18001 myself,master - 0 1533510957000 1 connected 0-5460
69655d72f2dca0419855b28f1fe26600ff35bc86 127.0.0.1:8006@18006 slave c4374d6f822233c0c7ae899caf4c59a254965f36 0 1533510957000 6 connected
88e2c0f6a29a1f15b5459ad4b8192aa81530c9f5 127.0.0.1:8005@18005 slave e48d4589b0197a34863b0a3567098fa45ed62a88 0 1533510958000 5 connected
c4374d6f822233c0c7ae899caf4c59a254965f36 127.0.0.1:8002@18002 master - 0 1533510958839 2 connected 5461-10922
d1c3c556fc2ca29053a11fa27dd86e537bcb7ca8 127.0.0.1:8004@18004 slave 5c4f74000b5923b21b7e62152ffd903ed6e299a7 0 1533510958537 4 connected
5c4f74000b5923b21b7e62152ffd903ed6e299a7 127.0.0.1:8003@18003 master - 0 1533510958000 3 connected 10923-16383

测试同上。

总结

手动实现 redis cluster 过程比较繁琐,但是有助于理解 redis cluster 的原理,通过节点握手、分配槽、配置从节点,最终完成了 redis cluster 的搭建。而 redis-trib 则提供了一个简单、易用的脚本以及一系列管理集群的命令,实际应用中正如 redis cluster 的 tutorial 所言,尽管你了解 redis-trib 内部是如何做的,你应该使用 redis-trib 来操作集群。

redis-trib 还提供了一些管理集群的命令,参见文档

-------------本文结束感谢阅读-------------
  • 本文标题:搭建 Redis 集群(手动、利用 redis-trib)
  • 本文作者:xlui
  • 发布时间:2018年08月05日 - 17:08
  • 最后更新:2018年09月05日 - 17:09
  • 本文链接: https://xlui.me/t/redis-cluster/
  • 版权声明: 本博客所有文章除特别声明外,均采用 CC BY-NC-ND 4.0 许可协议。转载请注明出处!