本文主要是介绍Redis 4集群failover后客户端未感知拓扑变化排查，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
排查结论
- 23：18分，30.1这台机器出现异常，其在30.2上的Slave Redis服务检测到后提出升主请求并获得通过，成为Master；
- 23：18至23：34之间，Redis client端对于Redis Server的连接检测并未异常，没有触发拓扑更新，因而对于Redis Server访问异常；
- 23：34，SRE重启30.1，此时 Redis Client检测到连接断开，触发拓扑更新，Redis访问正常。
解决方案
当前触发Redis Client更新拓扑的方式有两种：1. 向服务端发请求返回MOVE或ASK，2. ConnectionWatchDog检测到连接断开。当时的状况下，服务器状态未知（待与SRE确认），两种条件都没有触发。
所以解决方案为：Redis Client发新版，增加定期更新拓扑。
问题背景
2021-03-11 oncall群里由SRE发现一台redis服务器故障，通知如下
从而引发了帖子，蛋卷，UC，推荐相关服务的报警，redis的failover是在1min内完成的，但是其中推荐的服务的告警持续了近15min恢复，因此以下排查基于推荐服务进行
排查过程
时间线
redis服务器层面
| 节点 | 描述 |
|---|---|
| 30.1 | 物理机在23:34:59完成了服务启动 |
| 30.2 | 这个从节点在23:18:45完成了主从切换 |
| 30.1:
|
推荐服务层面
读写操作分析
| 节点 | 读写报错时间：start | 读写报错时间：end |
|---|---|---|
| 160-116 | 23:19:10开始出现读写报错 | 23:30:25读写报错结束（这个时间进行了服务重启） |
| 160-119 | 23:19:10开始出现读写报错 | 23:34:39读写报错结束（这个时间30.1那台redis物理机重新启动） |
| ------------------160-116 app@bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |tail ------------------160-119 app@bjxg-vd-160-119:~$ grep -B1 -A5 "io.lettuce.core.RedisCommandTimeoutException" /persist/logs/archives/stdout.2021-03-11.log |tail |
lettuce发现connection异常分析
| 节点 | 出现connection异常时间：start | 出现connection异常时间：end |
|---|---|---|
| 160-116 | 23:35:16出现connection异常提示 | 23:35:16出现connection异常提示，有且仅有这么一次（服务启动时发现拓扑的过程发生的错误） |
| 160-119 | 23:34:08出现connection异常提示（这个时间30.1那台redis物理机重新启动） | 23:42:35最后出现connection异常（23:42:26服务进行了重启） |
| ------------------160-116 app@bjxg-vd-160-116:/persist/logs/archives$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log |tail ------------------160-119 app@bjxg-vd-160-119:~$ grep -B1 -A5 ClusterTopologyRefresh /persist/logs/archives/stdout.2021-03-11.log |tail |
监控图层面
| 节点 | QPS异常开始时间 | QPS回复时间 |
|---|---|---|
| 160-116 | 23:19开始发生（与上面的读写操作日志分析对应上了） | 23:35开始恢复（与上面的lettuce发现connection日志异常分析对应上了，这个是重启的结果不能够说明SDK拓扑更新的逻辑） |
| 160-119 | 23:19开始发生（与上面的读写操作日志分析对应上了） | 23:34开始恢复（23:34服务没有重启，QPS回复正常，证明拓扑是更新了。 倾向认为：23：18~23：34 redis-server30.1不响应redis-Command，但是TCP未断连。 23：34由于TCP断连触发拓扑刷新） |
对应使用redis2.8的服务监控
帖子服务的2.8的redis client的P999是和上面吻合的
http://matrix.snowballfinance.com/d/ryrAxKXZk/ugc-snowflake-status-basic?orgId=1&from=1615474888737&to=1615478153783&fullscreen&panelId=8
UC服务的2.8的redis client的P999不吻合的（这个需要UC来解释）
http://matrix.snowballfinance.com/d/000000005/uc-snowflake-usercenter?orgId=1&fullscreen&edit&panelId=10&from=1615475760054&to=1615477519942
redis metric监控系统
redis的metric监控系统是golang编写的，原理就是for循环进行监控数据采集
| 时间 | 描述 |
|---|---|
| 在23:18的时候 | 对30.1这个机器的节点进行getinfo的时候，出现了阻塞 |
| 在23:34的时候 | 30.1这个机器重新启动，阻塞得到释放（从侧面印证了上述119这台机器监控和日志的解释） |
http://matrix.snowballfinance.com/d/000000024/sre-redis-detail?orgId=1&from=1615474845742&to=1615480482137
redis所在物理机监控
30.1物理机处于什么状态？导致18~34没有监控指标？
23：18-23：34监控数据没有，grafana使用了1条直线连接了23:18和23:34
http://matrix.snowballfinance.com/d/000000012/sre-wu-li-ji-detail?orgId=1&from=1615475297782&to=1615478502199&var-Hostname=bjxg-rs-30-1
原因分析
tcp/ip跟本次事件相关的原理背景
TCP 建立连接要进行三次握手（有对应的SYN信号），而断开连接要进行四次（有对应的FIN信号，且都是seq的序列号来进行确认）。这是由于TCP 的半关闭造成的。这也是为什么四次挥手要比三次握手多一次，因为TCP 连接是全双工的(即数据可在两个方向上同时传递)所以进行关闭时每个方向上都要单独进行关闭。单方向的关闭就叫半关闭。
TCP正常创建流程对fd的处理流程，TCP正常close和shutdown的处理流程，那么tcp连接异常终端后对应的socket会做些什么？
TCP/IP可选特性Keepalive（默认关闭）
有3个参数，以linux为例，搜索keepalive相关的系统参数（sysctl -A |grep keepalive），3个参数组合起来表示：
如果TCP连接2小时（7200秒）内没有数据传输，则发送嗅探包，每隔75秒发送一次，共重试9次。9次对方都没响应，则表明此连接已死。（留个问题，这个时候socket会给fd发送什么信号？？）
以linux为例：
recv的man对返回值的介绍如下：
RETURN VALUE
These calls return the number of bytes received, or -1 if an error occurred. The return value will be 0 when the peer has performed an orderly shutdown.
即，recv返回0表示对端关闭。对于send，如果返回值为-1，且errno为ECONNRESET，表示对端关闭。
一般服务器程序不会用到这个机制，而是把Keepalive放到业务层，自己实现Keepalive机制。这样做的好处是可以减少程序对特定协议的依赖，且发送给客户端的请求可以组合起来，在心跳包到来的时候一并返回给客户端。
摘自《TCP/IP详解》卷1第23章，强烈建议点开链接看看，解释的真通俗易懂
主要本次关注以下几点：
1.没有收到FIN信号，没有keepalive，那么该TCP会在两个小时之后尝试嗅探（天塌了它也不管）
2.对端服务器崩溃、关机、正在重启、网络不可达，那么该TCP的上层通常是报告连接超时（留个问题，信号量是多少？？）
3.对端服务重启之后，那么该TCP得到的回复是RST
netty跟本次事件相关的原理背景
netty这部分没啥好说的，看了好多的资料，总结下来关于本次事件相关的连接信息有以下内容：
1.NioEventLoop，reactor线程在自旋中执行细节
accept事件接收连接
2.workEventLoop，上一步的在接收完链接后，会触发相关读事件( fireChannelRead)，以及读完成事件( fireChannelReadComplete)对于server端来说，会设置对应selectKeys事件设置为 读事件监听。
由于会去pipeline中调用每个channelHandler进行事件处理，所以在nio 的accept事件之后，会执行 fireChannelRead 方法。
读事件第一个处理为 headContext。而后由 HeadContext 将事件进行向后传播。DefaultChannelPipeline 的 HeadContext。
3.close 与 channelInactive 的关系是 close 是主动关闭 channel 的动作, 而 channelInactive 是关闭成功后收到通知的事件.
关于netty的tcp参数设置
通过Netty配置TCP心跳，或者泛化的说配置TCP协议参数的方式有两种：
1、childOption方式是正确的
2、option方法不起作用，也不能说错吧，就是没啥意义而已
即本质就是一种方式。另外在4.1.36以后版本，如果你是JDK7以上，那么可以直接这样配置客户端连接的协议参数：.childOption(NioChannelOption.of(StandardSocketOptions.SO_KEEPALIVE),true)
网络层复现
背景介绍：使用的64.169/64.15这两台虚拟机用来当做服务器，本地开发机当做client
然后通过IPtable drop来尝试模拟服务节点突然下线
网络层的TCP信号
经历了开始的三次握手（SYN信号） ------→ 数据交互，正常读写（P.信号） ------→ 其中一个节点连接突然中断（R.信号） ------→ 尝试重连（不断的S信号） ------→ 连接恢复（SYN信号） ------→ 恢复正常读写（P.信号） ------→ 最后的链接断开（FIN信号）
20:08:39.861243 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [S], seq 1251904794, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731071045 ecr 0,sackOK,eol], length 0 20:08:39.861536 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [S.], seq 4226972078, ack 1251904795, win 28960, options [mss 1460,sackOK,TS val 107727801 ecr 731071045,nop,wscale 7], length 0 20:08:39.861574 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [.], ack 4226972079, win 2058, options [nop,nop,TS val 731071045 ecr 107727801], length 020:08:39.862538 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [P.], seq 1251904795:1251904861, ack 4226972079, win 2058, options [nop,nop,TS val 731071046 ecr 107727801], length 66 20:08:39.862838 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [.], ack 1251904861, win 227, options [nop,nop,TS val 107727802 ecr 731071046], length 0 20:08:39.863019 IP 192.168.64.15.8052 > 192.168.64.64.63823: Flags [P.], seq 4226972079:4226972084, ack 1251904861, win 227, options [nop,nop,TS val 107727802 ecr 731071046], length 5 20:08:39.863065 IP 192.168.64.64.63823 > 192.168.64.15.8052: Flags [.], ack 4226972084, win 2058, options [nop,nop,TS val 731071046 ecr 107727802], length 020:08:39.865550 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [S], seq 4104613940, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731071048 ecr 0,sackOK,eol], length 0 20:08:39.865820 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [S.], seq 413298472, ack 4104613941, win 28960, options [mss 1460,sackOK,TS val 107727805 ecr 731071048,nop,wscale 7], length 0 20:08:39.865868 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [.], ack 413298473, win 2058, options [nop,nop,TS val 731071049 ecr 107727805], length 020:08:39.866943 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [P.], seq 4104613941:4104614007, ack 413298473, win 2058, options [nop,nop,TS val 731071050 ecr 107727805], length 66 20:08:39.867192 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [.], ack 4104614007, win 227, options [nop,nop,TS val 107727807 ecr 731071050], length 0 20:08:39.867329 IP 192.168.64.15.8051 > 192.168.64.64.63824: Flags [P.], seq 413298473:413298478, ack 4104614007, win 227, options [nop,nop,TS val 107727807 ecr 731071050], length 5 20:08:39.867378 IP 192.168.64.64.63824 > 192.168.64.15.8051: Flags [.], ack 413298478, win 2058, options [nop,nop,TS val 731071050 ecr 107727807], length 014 20:16:00.604659 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509045 ecr 107733081], length 48 20:16:00.754648 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509195 ecr 107733081], length 48 20:16:01.055889 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509495 ecr 107733081], length 48 20:16:01.456264 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731509895 ecr 107733081], length 48 20:16:02.062739 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731510495 ecr 107733081], length 48 20:16:03.065437 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731511495 ecr 107733081], length 48 20:16:04.871576 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731513295 ecr 107733081], length 48 20:16:08.287194 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731516695 ecr 107733081], length 48 20:16:14.912828 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731523295 ecr 107733081], length 48 20:16:21.529555 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731529895 ecr 107733081], length 48 20:16:28.147866 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731536495 ecr 107733081], length 48 20:16:34.771964 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731543095 ecr 107733081], length 48 20:16:41.398904 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731549695 ecr 107733081], length 48 20:16:48.025246 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [P.], seq 4241911372:4241911420, ack 1651645875, win 2048, options [nop,nop,TS val 731556295 ecr 107733081], length 48 20:16:54.655115 IP 192.168.64.64.63832 > 192.168.64.15.8051: Flags [R.], seq 4241911420, ack 1651645875, win 2048, length 07 20:17:17.609293 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731585757 ecr 0,sackOK,eol], length 0 20:17:18.610567 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731586757 ecr 0,sackOK,eol], length 0 20:17:19.613758 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731587757 ecr 0,sackOK,eol], length 0 20:17:20.617512 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731588758 ecr 0,sackOK,eol], length 0 20:17:21.620555 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731589758 ecr 0,sackOK,eol], length 0 20:17:22.625515 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731590758 ecr 0,sackOK,eol], length 0 20:17:24.631276 IP 192.168.64.64.64283 > 192.168.64.15.8051: Flags [S], seq 2179805839, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731592758 ecr 0,sackOK,eol], length 011 20:17:27.661985 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731595780 ecr 0,sackOK,eol], length 0 20:17:27.805463 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731595921 ecr 0,sackOK,eol], length 0 20:17:27.906594 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596021 ecr 0,sackOK,eol], length 0 20:17:28.007657 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596122 ecr 0,sackOK,eol], length 0 20:17:28.108495 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596222 ecr 0,sackOK,eol], length 0 20:17:28.211396 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596322 ecr 0,sackOK,eol], length 0 20:17:28.412573 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596523 ecr 0,sackOK,eol], length 0 20:17:28.813800 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731596923 ecr 0,sackOK,eol], length 0 20:17:29.617595 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731597723 ecr 0,sackOK,eol], length 0 20:17:31.226228 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731599323 ecr 0,sackOK,eol], length 0 20:17:34.434295 IP 192.168.64.64.64291 > 192.168.64.15.8051: Flags [S], seq 2690785073, win 65535, options [mss 1460,sackOK,eol], length 011 20:17:37.766336 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731605835 ecr 0,sackOK,eol], length 0 20:17:37.907801 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731605976 ecr 0,sackOK,eol], length 0 20:17:38.009930 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606076 ecr 0,sackOK,eol], length 0 20:17:38.111023 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606177 ecr 0,sackOK,eol], length 0 20:17:38.212162 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606277 ecr 0,sackOK,eol], length 0 20:17:38.312800 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606377 ecr 0,sackOK,eol], length 0 20:17:38.513278 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606577 ecr 0,sackOK,eol], length 0 20:17:38.913743 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731606977 ecr 0,sackOK,eol], length 0 20:17:39.718061 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731607777 ecr 0,sackOK,eol], length 0 20:17:41.323608 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731609377 ecr 0,sackOK,eol], length 0 20:17:44.538196 IP 192.168.64.64.64302 > 192.168.64.15.8051: Flags [S], seq 3980797911, win 65535, options [mss 1460,sackOK,eol], length 011 20:17:47.867437 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731615897 ecr 0,sackOK,eol], length 0 20:17:48.008751 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616038 ecr 0,sackOK,eol], length 0 20:17:48.109361 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616138 ecr 0,sackOK,eol], length 0 20:17:48.209657 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616238 ecr 0,sackOK,eol], length 0 20:17:48.312000 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616338 ecr 0,sackOK,eol], length 0 20:17:48.413063 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616439 ecr 0,sackOK,eol], length 0 20:17:48.613676 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731616639 ecr 0,sackOK,eol], length 0 20:17:49.015418 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731617039 ecr 0,sackOK,eol], length 0 20:17:49.818549 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731617839 ecr 0,sackOK,eol], length 0 20:17:51.422985 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731619439 ecr 0,sackOK,eol], length 0 20:17:54.631568 IP 192.168.64.64.64311 > 192.168.64.15.8051: Flags [S], seq 4094832470, win 65535, options [mss 1460,sackOK,eol], length 011 20:17:57.969994 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731625971 ecr 0,sackOK,eol], length 0 20:17:57.970906 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731625971 ecr 0,sackOK,eol], length 0 20:17:58.115551 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626112 ecr 0,sackOK,eol], length 0 20:17:58.115561 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626112 ecr 0,sackOK,eol], length 0 20:17:58.217038 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626212 ecr 0,sackOK,eol], length 0 20:17:58.217049 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626212 ecr 0,sackOK,eol], length 0 20:17:58.317493 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626312 ecr 0,sackOK,eol], length 0 20:17:58.317504 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626312 ecr 0,sackOK,eol], length 0 20:17:58.418207 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626412 ecr 0,sackOK,eol], length 0 20:17:58.418219 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626412 ecr 0,sackOK,eol], length 0 20:17:58.519006 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626512 ecr 0,sackOK,eol], length 0 20:17:58.519018 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626512 ecr 0,sackOK,eol], length 0 20:17:58.721709 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626712 ecr 0,sackOK,eol], length 0 20:17:58.721721 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731626712 ecr 0,sackOK,eol], length 0 20:17:59.123040 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627112 ecr 0,sackOK,eol], length 0 20:17:59.123052 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627112 ecr 0,sackOK,eol], length 0 20:17:59.925894 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627912 ecr 0,sackOK,eol], length 0 20:17:59.925910 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731627912 ecr 0,sackOK,eol], length 0 20:18:01.530428 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731629512 ecr 0,sackOK,eol], length 0 20:18:01.530436 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731629512 ecr 0,sackOK,eol], length 0 20:18:04.739272 IP 192.168.64.64.64320 > 192.168.64.15.8051: Flags [S], seq 1392414358, win 65535, options [mss 1460,sackOK,eol], length 0 20:18:04.739280 IP 192.168.64.64.64319 > 192.168.64.15.8051: Flags [S], seq 3138653942, win 65535, options [mss 1460,sackOK,eol], length 013 20:18:08.063149 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636018 ecr 0,sackOK,eol], length 0 20:18:08.204712 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636159 ecr 0,sackOK,eol], length 0 20:18:08.305275 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636259 ecr 0,sackOK,eol], length 0 20:18:08.406168 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636359 ecr 0,sackOK,eol], length 0 20:18:08.507152 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636459 ecr 0,sackOK,eol], length 0 20:18:08.607218 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636559 ecr 0,sackOK,eol], length 0 20:18:08.808894 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731636759 ecr 0,sackOK,eol], length 0 20:18:09.209305 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731637159 ecr 0,sackOK,eol], length 0 20:18:10.011676 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731637959 ecr 0,sackOK,eol], length 0 20:18:11.616529 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731639559 ecr 0,sackOK,eol], length 0 20:18:14.828898 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 0 20:18:21.286606 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 0 20:18:27.721777 IP 192.168.64.64.64335 > 192.168.64.15.8051: Flags [S], seq 3704085195, win 65535, options [mss 1460,sackOK,eol], length 013 20:20:58.899104 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731805899 ecr 0,sackOK,eol], length 0 20:20:58.900162 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731805900 ecr 0,sackOK,eol], length 0 20:20:59.045755 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806040 ecr 0,sackOK,eol], length 0 20:20:59.047010 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806041 ecr 0,sackOK,eol], length 0 20:20:59.146241 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806140 ecr 0,sackOK,eol], length 0 20:20:59.147300 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806141 ecr 0,sackOK,eol], length 0 20:20:59.247515 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806240 ecr 0,sackOK,eol], length 0 20:20:59.248746 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806241 ecr 0,sackOK,eol], length 0 20:20:59.347871 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806340 ecr 0,sackOK,eol], length 0 20:20:59.348974 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806341 ecr 0,sackOK,eol], length 0 20:20:59.449197 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806441 ecr 0,sackOK,eol], length 0 20:20:59.449211 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806441 ecr 0,sackOK,eol], length 0 20:20:59.654647 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806641 ecr 0,sackOK,eol], length 0 20:20:59.654653 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731806641 ecr 0,sackOK,eol], length 0 20:21:00.056939 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807041 ecr 0,sackOK,eol], length 0 20:21:00.056958 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807041 ecr 0,sackOK,eol], length 0 20:21:00.867762 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807842 ecr 0,sackOK,eol], length 0 20:21:00.867778 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731807842 ecr 0,sackOK,eol], length 0 20:21:02.484895 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731809442 ecr 0,sackOK,eol], length 0 20:21:02.484903 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731809442 ecr 0,sackOK,eol], length 0 20:21:05.715910 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0 20:21:05.715916 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 0 20:21:12.139150 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0 20:21:12.139164 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 0 20:21:18.570932 IP 192.168.64.64.64522 > 192.168.64.15.8051: Flags [S], seq 442060788, win 65535, options [mss 1460,sackOK,eol], length 0 20:21:18.570946 IP 192.168.64.64.64519 > 192.168.64.15.8051: Flags [S], seq 171452135, win 65535, options [mss 1460,sackOK,eol], length 020:23:49.944343 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [S], seq 1827331675, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731976206 ecr 0,sackOK,eol], length 0 20:23:49.944875 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [S], seq 1554881183, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 731976207 ecr 0,sackOK,eol], length 0 20:23:49.954078 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [S.], seq 2130024761, ack 1827331676, win 28960, options [mss 1460,sackOK,TS val 108637891 ecr 731976206,nop,wscale 7], length 0 20:23:49.954085 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [S.], seq 1062682904, ack 1554881184, win 28960, options [mss 1460,sackOK,TS val 108637891 ecr 731976207,nop,wscale 7], length 0 20:23:49.954203 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [.], ack 2130024762, win 2058, options [nop,nop,TS val 731976215 ecr 108637891], length 0 20:23:49.954257 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [.], ack 1062682905, win 2058, options [nop,nop,TS val 731976215 ecr 108637891], length 0 20:23:49.957101 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [P.], seq 1827331676:1827331724, ack 2130024762, win 2058, options [nop,nop,TS val 731976217 ecr 108637891], length 48 20:23:49.959283 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881184:1554881250, ack 1062682905, win 2058, options [nop,nop,TS val 731976218 ecr 108637891], length 66 20:23:49.999821 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [.], ack 1827331724, win 227, options [nop,nop,TS val 108637944 ecr 731976217], length 0 20:23:49.999822 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881250, win 227, options [nop,nop,TS val 108637944 ecr 731976218], length 0 20:23:50.000112 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881250:1554881278, ack 1062682905, win 2058, options [nop,nop,TS val 731976258 ecr 108637944], length 28 20:23:50.000541 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881278, win 227, options [nop,nop,TS val 108637944 ecr 731976258], length 0 20:23:50.001223 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [P.], seq 1554881278:1554881304, ack 1062682905, win 2058, options [nop,nop,TS val 731976259 ecr 108637944], length 26 20:23:50.001880 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [.], ack 1554881304, win 227, options [nop,nop,TS val 108637946 ecr 731976259], length 0 20:23:50.002150 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [P.], seq 1062682905:1062683695, ack 1554881304, win 227, options [nop,nop,TS val 108637946 ecr 731976259], length 79020:40:12.418726 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [F.], seq 1554881304, ack 1062684496, win 2048, options [nop,nop,TS val 732912682 ecr 108637947], length 0 20:40:12.418803 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [F.], seq 1827331724, ack 2130024795, win 2058, options [nop,nop,TS val 732912682 ecr 108637946], length 0 20:40:12.421643 IP 192.168.64.15.8051 > 192.168.64.64.64671: Flags [F.], seq 1062684496, ack 1554881305, win 227, options [nop,nop,TS val 109620371 ecr 732912682], length 0 20:40:12.421695 IP 192.168.64.64.64671 > 192.168.64.15.8051: Flags [.], ack 1062684497, win 2048, options [nop,nop,TS val 732912685 ecr 109620371], length 0 20:40:12.422976 IP 192.168.64.15.8051 > 192.168.64.64.64670: Flags [F.], seq 2130024795, ack 1827331725, win 227, options [nop,nop,TS val 109620372 ecr 732912682], length 0 20:40:12.423014 IP 192.168.64.64.64670 > 192.168.64.15.8051: Flags [.], ack 2130024796, win 2058, options [nop,nop,TS val 732912686 ecr 109620372], length 0 |
中间过程发现在进行topology刷新的时候出现大于1的的TCP连接，这些多余的连接都发生在问题节点所在的ip上，clientName叫做：lettuce#ClusterTopologyRefresh
[root@localhost ~]# /data/server/redis-cluster/redis-5.0.5/src/redis-cli -c -h 192.168.64.15 -p 8052 client list |grep 64.64 id=287 addr=192.168.64.64:64321 fd=33 name=lettuce#ClusterTopologyRefresh age=1292 idle=1282 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client id=289 addr=192.168.64.64:64523 fd=35 name=lettuce#ClusterTopologyRefresh age=1111 idle=943 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client id=290 addr=192.168.64.64:64672 fd=36 name=lettuce#ClusterTopologyRefresh age=940 idle=940 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=client id=286 addr=192.168.64.64:63831 fd=13 name= age=1849 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=5 oll=0 omem=0 events=r cmd=setex |
Netty层的channel信号
中间过程当第一次出现读写超时事件时，netty是能够收到inactive事件（虽然这个时间点没有接收到TCP的对应信号，那么netty是怎么判断关闭TCP的呢？答： [R.] 就是这个reset信号量，这个就是最后的倔强）
然后就会触发scheduleReconnect方法（后续会阐述这个重连实现）尝试重连，这个过程中redis cluster正在进行主从切换，挡住从切换完成之后client端获取到正常的网络拓扑，读写恢复正常。
但是读写就算恢复正常，失去连接的节点底层还是会触发inactive事件，所以还是一直会报io.netty.channel.ConnectTimeoutException: connection timed out:/192.168.64.15:8052异常
java.util.concurrent.CompletionException: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.64.15:8052at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_201]at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.6.RELEASE.jar:na]at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:269) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201] Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /192.168.64.15:8052at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]... 9 common frames omitted 20:58:19.845 WARN i.l.c.c.t.ClusterTopologyRefresh - Unable to connect to 192.168.64.13:8051 java.util.concurrent.CompletionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.64.13:8051at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_201]at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_201]at io.lettuce.core.AbstractRedisClient.lambda$initializeChannelAsync0$4(AbstractRedisClient.java:330) ~[lettuce-core-5.1.6.RELEASE.jar:na]at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.33.Final.jar:4.1.33.Final]at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_201] Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.64.13:8051at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_201]at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_201]at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[netty-transport-4.1.27.Final.jar:4.1.27.Final]... 7 common frames omitted Caused by: java.net.ConnectException: Connection refused... 11 common frames omitted |
复现不出来，好痛苦，接下来就看看这个TCP的reset信号啥时候发送，以及是netty触发的还是TCP触发的。。。
这篇关于Redis 4集群failover后客户端未感知拓扑变化排查的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
