本文主要是介绍redis cluster topology handshake问题，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
问题背景：
2021-01-29 由于55.6物理机可用内存较少，担心内存在高峰期吃紧对社区帖子服务的comment集群进行了节点的迁移（在迁移之前帖子服务redis4的client连接已经构建，只是没有读写流量）
1.下线了2个实例 10.10.55.6:26962和10.10.55.6:26966，其中一个是master，另外一个是slave
2.上线了2个实例 10.10.164.15:26962和10.10.164.15:26966，对上一步的两个节点进行替换
2021-02-20 帖子服务打开了redis4的读写流量，此时发现error报错
错误内容显示帖子服务还是一直访问的10.10.55.6:26962这个已经被下掉替换的节点（预期的访问是10.10.164.15:26962这个节点）
问题复盘：
case1：
1.集群3主3从，分别是：A-A1、B-B1、C-C1（此时client连接建立，无读写）
2.将A节点替换成新的节点D，新的集群是：D-A1、B-B1、C-C1
3.替换之后打开读写
结果：client能够获取到server的拓扑信息，不能复现
case2：
1.集群3主3从，分别是：A、B、C和A1、B1、C1（此时client连接建立，无读写）
2.在B节点将A节点从集群forget，然后在A节点执行shutdown（此时模拟问题场景）
3.操作后的集群是：A(handshake)-A1、B-B1、C-C1
4.之后打开读写
结果：client访问报错，问题复现
原因分析：
其中在node的添加和摘除操作中涉及到：
CLUSTER MEET ip port
CLUSTER FORGET node-id
这两个命令，在MEET操作中，会进行socket通信的建立和协议的交换，其中有个中间过程叫做handshake
在redis手册中我们能够看到：
https://redis.io/commands/cluster-meet
An already known node sends a list of nodes in the gossip section that we are not aware of. If the receiving node trusts the sending node as a known node, it will process the gossip section and send an handshake to the nodes that are still not known.
简述：在meet操作时，会有handshake动作，具体实现见：clusterStartHandshake函数
https://redis.io/commands/cluster-forget
Details on why the ban-list is needed
In the following example we'll show why the command must not just remove a given node from the nodes table, but also prevent it for being re-inserted again for some time.
Let's assume we have four nodes, A, B, C and D. In order to end with just a three nodes cluster A, B, C we may follow these steps:
- Reshard all the hash slots from D to nodes A, B, C.
- D is now empty, but still listed in the nodes table of A, B and C.
- We contact A, and send
CLUSTER FORGET D. - B sends node A a heartbeat packet, where node D is listed.
- A does no longer known node D (see step 3), so it starts an handshake with D.
- D ends re-added in the nodes table of A.
简述：其中在进行forget之后，其他的节点gossip还在进行，会将forget掉的那个节点信息传播进来，这时候执行forget命令的节点就会发生handshake动作，尝试连接这个被forget掉的节点（注意：这个时候要是那个forget的节点已经shutdown了，那么当前的handshake将会如何处理？）
Special conditions not allowing the command execution
The command does not succeed and returns an error in the following cases:
- The specified node ID is not found in the nodes table.
- The node receiving the command is a replica, and the specified node ID identifies its current master.
- The node ID identifies the same node we are sending the command to.
简述：其中列了三个case会导致forget执行失败（nodeID不存在，nodeID是自己，nodeID是自己的master）
在redis的issue里面我们能够找到：
https://github.com/antirez/redis/issues/2965
There are only two ways this can happen:
- You fail to send CLUSTER FORGET to all the nodes in the cluster. So eventually there are nodes that still has a clue about this other node, and it will inform the other nodes via gossip. Make sure to send CLUSTER FORGET to every single node in the cluster.
- Or alternatively, there is an instance running in 10.15.107.150 but you said there is not.
在进行forget的时候，没有确认所有的node都成功执行了forget命令，导致gossip下次广播的时候将下掉的节点信息带到了forget的节点上（注意：已经下掉的node收到这个gossip就会尝试进行meet，那么就会出现handshake动作）
至于为什么不下掉，引自下面资料源码解释：
/* If it's not in NOADDR state and we don't have it, we* start a handshake process against this IP/PORT pairs.** Note that we require that the sender of this gossip message* is a well known node in our cluster, otherwise we risk* joining another cluster. */if (sender &&!(flags & CLUSTER_NODE_NOADDR) &&!clusterBlacklistExists(g->nodename)){clusterStartHandshake(g->ip,ntohs(g->port));}搜索代码中改变nodeid定位到createClusterNode()，定位到其调用过程clusterStartHandshake()->createClusterNode()->getRandomHexChars(node->name, CLUSTER_NAMELEN)随机生成nodeid，进入handshake状态. (如果下线的节点重新上线了，与该节点成功建立连接，并在收到该节点报文后更新其nodeid为节点真正的nodeid)结合forget过程中强调的nodeID不一致会导致下不掉就不难分析出原因改进措施：
既然handshake状态是由于收到fail状态信息导致的，那么只用把fail状态forget掉就可以，而且fail状态的节点A nodeid是一直不变的.
在集群的每个节点，执行cluster forget其节点包含fail状态节点的nodeid, 之后handshake状态信息也不见了，再次执行数据扩容操作，成功完成.
issue提供的解决脚本：
#echo "usage: host port"
nodes_addrs=$(redis-cli -h $1 -p $2 cluster nodes|grep -v handshake| awk '{print $2}')
echo $nodes_addrs
for addr in ${nodes_addrs[@]}; dohost=${addr%:*}port=${addr#*:}del_nodeids=$(redis-cli -h $host -p $port cluster nodes|grep -E 'handshake|fail'| awk '{print $1}')for nodeid in ${del_nodeids[@]}; doecho $host $port $nodeidredis-cli -h $host -p $port cluster forget $nodeiddone
done添加过程中程序会遍历集群各节点执行cluster info命令检查cluster_known_nodes是否达到预期结果
参考资料：
handshake源码讲解：https://www.cnblogs.com/gqtcgq/p/7247044.html
handshake例子解析：https://githubmota.github.io/2018/06/15/TODO/
redis cluster的集群通信是通过gossip进行的，关于gossip的介绍：
https://singgel.blog.csdn.net/article/details/107489947
https://singgel.blog.csdn.net/article/details/107490037
这篇关于redis cluster topology handshake问题的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
