为什么通过IPVS访问服务，响应包没有转换IP，而是直接发给了client？

Question

我的环境是一个kubeedge集群，因为一些原因，边缘节点上没有跑kube-proxy，而是用我们自己的agent生成IPVS配置。

出问题的节点叫edge1, 上面跑了两个Pod： net-tool-edge1和nginx-edge1。net-tool-edge1是客户端， nginx-edge1是服务端，还有个叫nginx的service, nginx-edge1就是nginx的一个endpoint.

IP地址分配如下：

net-tool-edge1: 10.234.67.29
nginx-edge1: 10.234.67.29
nginx clusterIP: 10.234.39.157

今天发现net-tool-edge1不能访问nginx了，用tcpdump抓包发现net-tool-edge1收到的回报源地址不是10.234.39.157而是10.234.67.29，即nginx-edge1响应的时候是直接发给net-tool-edge1了，所以net-tool-edge1每次都会reset链接导致通信失败。

net-tool-edge1上的抓包结果如下：

02:45:46.016748 ARP, Request who-has 10.234.67.1 tell 10.234.67.29, length 28
02:45:46.016858 ARP, Request who-has 10.234.67.1 tell 10.234.67.29, length 28
02:45:46.016862 ARP, Reply 10.234.67.1 is-at 96:b9:e1:32:f0:fa, length 28
02:45:46.016864 IP 10.234.67.29.52704 > 169.254.25.10.53: 6768+ A? nginx.fabedge-e2e-test.svc.cluster.local. (58)
02:45:46.016953 IP 10.234.67.29.52704 > 169.254.25.10.53: 7300+ AAAA? nginx.fabedge-e2e-test.svc.cluster.local. (58)
02:45:46.025844 IP 169.254.25.10.53 > 10.234.67.29.52704: 6768*- 1/0/0 A 10.234.39.157 (114)
02:45:47.023403 IP 169.254.25.10.53 > 10.234.67.29.52704: 7300*- 0/1/0 (151)
02:45:47.023958 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 688253 ecr 0,nop,wscale 7], length 0
02:45:47.024040 ARP, Request who-has 10.234.67.28 tell 10.234.67.1, length 28
02:45:47.024149 ARP, Request who-has 10.234.67.29 tell 10.234.67.28, length 28
02:45:47.024153 ARP, Reply 10.234.67.29 is-at f2:3e:7d:a6:f5:1d, length 28
02:45:47.024162 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3004459791, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 688253 ecr 688253,nop,wscale 7], length 0
02:45:47.024180 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
02:45:48.026571 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 689256 ecr 0,nop,wscale 7], length 0
02:45:48.026687 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3020124674, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 689256 ecr 689256,nop,wscale 7], length 0
02:45:48.026702 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
02:45:51.026582 ARP, Request who-has 10.234.67.29 tell 10.234.67.1, length 28
02:45:51.026595 ARP, Reply 10.234.67.29 is-at f2:3e:7d:a6:f5:1d, length 28
02:45:52.034599 ARP, Request who-has 10.234.67.28 tell 10.234.67.29, length 28
02:45:52.034668 ARP, Reply 10.234.67.28 is-at 92:01:73:4f:50:2f, length 28
02:48:30.719257 IP6 fe80::cc0:62ff:fe90:582a > ff02::1:ff01:301: ICMP6, neighbor solicitation, who has fd96:ee88:d8a6:8607::1:301, length 32

在主机edge1上抓包结果如下:

[root@edge1 ~]# tcpdump -nn -i any port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
10:45:47.023961 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 688253 ecr 0,nop,wscale 7], length 0
10:45:47.023978 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 688253 ecr 0,nop,wscale 7], length 0
10:45:47.024063 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 688253 ecr 0,nop,wscale 7], length 0
10:45:47.024064 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 688253 ecr 0,nop,wscale 7], length 0
10:45:47.024160 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3004459791, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 688253 ecr 688253,nop,wscale 7], length 0
10:45:47.024161 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3004459791, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 688253 ecr 688253,nop,wscale 7], length 0
10:45:47.024185 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
10:45:47.024186 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
10:45:48.026585 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 689256 ecr 0,nop,wscale 7], length 0
10:45:48.026598 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 689256 ecr 0,nop,wscale 7], length 0
10:45:48.026636 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 689256 ecr 0,nop,wscale 7], length 0
10:45:48.026640 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 689256 ecr 0,nop,wscale 7], length 0
10:45:48.026684 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3020124674, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 689256 ecr 689256,nop,wscale 7], length 0
10:45:48.026686 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3020124674, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 689256 ecr 689256,nop,wscale 7], length 0
10:45:48.026703 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
10:45:48.026704 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0

可以看出，net-work-edge1(10.234.67.29)的发出的包目标地址是10.234.39.157，然后经过dnat转换，变成了10.234.67.29.57824 > 10.234.67.28，但nginx-edge1 回包的时候缺少了转换源地址的步骤，这导致net-tool-edge1直接收到nginx-edge1的回包。

在网上搜了好久，有人说需要做SNAT，但我另一个机器的配置跟edge1都是一样的，实际上edge1之前也是工作正常的，只是重启后出现了问题。现在没有任何头绪，只好上网求助了。

希望获得大家的帮助，先谢过了。

还有一些iptables和ipset的配置，贴在后面:

[root@edge1 ~]# iptables -S 
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N FABEDGE-FORWARD
-A INPUT -d 169.254.25.10/32 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -d 169.254.25.10/32 -p tcp -m tcp --dport 53 -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j FABEDGE-FORWARD
-A OUTPUT -s 169.254.25.10/32 -p udp -m udp --sport 53 -j ACCEPT
-A OUTPUT -s 169.254.25.10/32 -p tcp -m tcp --sport 53 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A FABEDGE-FORWARD -s 10.234.67.0/24 -j ACCEPT
-A FABEDGE-FORWARD -d 10.234.67.0/24 -j ACCEPT

[root@edge1 ~]# iptables -t nat -S 
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N FABEDGE-NAT-OUTGOING
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -j FABEDGE-NAT-OUTGOING
-A DOCKER -i docker0 -j RETURN
-A FABEDGE-NAT-OUTGOING -s 10.234.67.0/24 -m set --match-set FABEDGE-PEER-CIDR dst -j RETURN
-A FABEDGE-NAT-OUTGOING -s 10.234.67.0/24 -d 10.234.67.0/24 -j RETURN
-A FABEDGE-NAT-OUTGOING -s 10.234.67.0/24 -j MASQUERADE

 ipset list 
Name: FABEDGE-PEER-CIDR
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 952
References: 1
Number of entries: 9
Members:
10.234.66.0/24
10.234.0.0/18
10.22.48.28
10.22.48.16
10.22.48.34
10.234.64.0/24
10.234.68.0/24
10.234.65.0/24
10.22.48.17

Answer 1

破案了，有个叫net.bridge.bridge-nf-call-iptables的系统参数在节点重启后变成0了。

主要原因是从net-tool-edge1访问nginx的时候，在edge1会做nat，这个过程发生在3层,会生成一些数据记录在conntrack表里，这样回包的时候就可以再做一次nat。

当nginx-edge1发送回包的时候，因为两个pod都在一个节点，会直接通过一个网桥转发，这个过程发生在2层，所以不会去参考conntrack，也就没有nat发生。

更详细的解释看这篇文章：为什么 kubernetes 环境要求开启 bridge-nf-call-iptables ?