极客观点
项目管理
HarmonyOS
网络拓扑图
阿里云节点环境
- 操作系统：ubuntu 22.04LTS
网络配置：
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:16:3e:08:3c:50 brd ff:ff:ff:ff:ff:ff altname enp0s3 altname ens3 inet 172.19.27.60/20 metric 100 brd 172.19.31.255 scope global dynamic eth0 valid_lft 295876523sec preferred_lft 295876523sec inet6 fe80::216:3eff:fe08:3c50/64 scope link valid_lft forever preferred_lft forever 38: netmaker: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet 192.168.10.1/24 brd 192.168.10.255 scope global netmaker valid_lft forever preferred_lft forever路由信息：
default via 172.19.31.253 dev eth0 proto dhcp src 172.19.27.60 metric 100 100.100.2.136 via 172.19.31.253 dev eth0 proto dhcp src 172.19.27.60 metric 100 100.100.2.138 via 172.19.31.253 dev eth0 proto dhcp src 172.19.27.60 metric 100 172.19.16.0/20 dev eth0 proto kernel scope link src 172.19.27.60 metric 100 172.19.31.253 dev eth0 proto dhcp scope link src 172.19.27.60 metric 100 192.168.10.0/24 dev netmaker proto kernel scope link src 192.168.10.1 192.168.31.0/24 via 192.168.10.1 dev netmaker办公室节点环境
- 操作系统：openwrt 23.05
网络配置：
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000 link/sit 0.0.0.0 brd 0.0.0.0 3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP qlen 1000 link/ether 00:e0:4c:69:1a:62 brd ff:ff:ff:ff:ff:ff 4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 30:9c:23:a3:db:56 brd ff:ff:ff:ff:ff:ff inet6 fe80::329c:23ff:fea3:db56/64 scope link valid_lft forever preferred_lft forever 24: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000 link/ether 00:e0:4c:69:1a:62 brd ff:ff:ff:ff:ff:ff inet 192.168.31.1/24 brd 192.168.31.255 scope global br-lan valid_lft forever preferred_lft forever inet6 240e:370:4312:eee0::1/64 scope global dynamic noprefixroute valid_lft 256217sec preferred_lft 169817sec inet6 fe80::2e0:4cff:fe69:1a62/64 scope link valid_lft forever preferred_lft forever 25: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN qlen 3 link/ppp inet 100.64.215.177 peer 100.64.128.1/32 scope global pppoe-wan valid_lft forever preferred_lft forever inet6 240e:370:4312:eee1::1/64 scope global dynamic noprefixroute valid_lft 256217sec preferred_lft 169817sec inet6 240e:370:4211:c759:997:9f43:7f6:e31/64 scope global dynamic noprefixroute valid_lft 258851sec preferred_lft 172451sec inet6 fe80::997:9f43:7f6:e31/128 scope link valid_lft forever preferred_lft forever 26: netmaker: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1420 qdisc fq_codel state UNKNOWN qlen 500 link/[65534] inet 192.168.10.3/24 brd 192.168.10.255 scope global netmaker valid_lft forever preferred_lft forever防火墙配置：
config defaults option input 'ACCEPT' option output 'ACCEPT' option forward 'ACCEPT' option fullcone '1' option synflood_protect '1' option drop_invalid '1' config zone option name 'lan' option input 'ACCEPT' option output 'ACCEPT' option forward 'ACCEPT' list network 'lan' config zone option name 'dmz' option input 'REJECT' option output 'ACCEPT' option forward 'ACCEPT' option auto_helper '0' option log '1' list network 'netmaker' config zone option name 'wan' option input 'REJECT' option output 'ACCEPT' option forward 'REJECT' option masq '1' option mtu_fix '1' option log '1' list network 'wan' list network 'wan6' config rule option name 'Allow-DHCP-Renew' option src 'wan' option proto 'udp' option dest_port '68' option target 'ACCEPT' option family 'ipv4' config rule option name 'Allow-Ping' option src 'wan' option proto 'icmp' option icmp_type 'echo-request' option family 'ipv4' option target 'ACCEPT' config rule option name 'Allow-IGMP' option src 'wan' option proto 'igmp' option family 'ipv4' option target 'ACCEPT' config rule option name 'Allow-DHCPv6' option src 'wan' option proto 'udp' option dest_port '546' option family 'ipv6' option target 'ACCEPT' config rule option name 'Allow-MLD' option src 'wan' option proto 'icmp' option src_ip 'fe80::/10' list icmp_type '130/0' list icmp_type '131/0' list icmp_type '132/0' list icmp_type '143/0' option family 'ipv6' option target 'ACCEPT' config rule option name 'Allow-ICMPv6-Input' option src 'wan' option proto 'icmp' option family 'ipv6' option target 'ACCEPT' list icmp_type 'bad-header' list icmp_type 'destination-unreachable' list icmp_type 'echo-reply' list icmp_type 'echo-request' list icmp_type 'neighbour-advertisement' list icmp_type 'neighbour-solicitation' list icmp_type 'packet-too-big' list icmp_type 'router-advertisement' list icmp_type 'router-solicitation' list icmp_type 'time-exceeded' list icmp_type 'unknown-header-type' config rule option name 'Allow-ICMPv6-Forward' option src 'wan' option dest '*' option proto 'icmp' option family 'ipv6' option target 'ACCEPT' list icmp_type 'bad-header' list icmp_type 'destination-unreachable' list icmp_type 'echo-reply' list icmp_type 'echo-request' list icmp_type 'packet-too-big' list icmp_type 'time-exceeded' list icmp_type 'unknown-header-type' config forwarding option src 'lan' option dest 'wan' config forwarding option src 'wan' option dest 'lan' config forwarding option src 'dmz' option dest 'lan' config forwarding option src 'lan' option dest 'dmz'路由信息：
default via 100.64.128.1 dev pppoe-wan 100.64.128.1 dev pppoe-wan scope link src 100.64.215.177 172.19.0.0/16 via 192.168.10.3 dev netmaker 192.168.10.0/24 dev netmaker scope link src 192.168.10.3 192.168.31.0/24 dev br-lan scope link src 192.168.31.1问题
在阿里云主机 172.19.227.25 上安装 mysql 服务，端口为 30000。
通过以下几种方式执行 mysql -uroot -p'xxxx' -P30000 -h172.19.227.25 命令都能正
常的连接并返回“Server version: 5.7.43-log MySQL Community Server (GPL)”响应。
但是继续执行 use mysql; 命令时：
| 主机 | 结果 |
|---|---|
| 192.168.31.1（openwrt主机） | 成功 |
| 192.168.31.160（局域网的另一台主机） | 无响应，命令行 hang 住了 |
| 192.168.31.222（安装了 netclient 的主机） | 成功 |
抓包日志
- 192.168.31.160 丢包日志：https://tmp-repo.oss-cn-shanghai.aliyuncs.com/mysql.cap
192.168.31.222 正常日志：https://tmp-repo.oss-cn-shanghai.aliyuncs.com/mysql3.cap
其他
我也试过在 192.168.31.160 上部署 mysql 服务，能从阿里云的 172.19.227.25 上正常访问。
测试下来只有 yintan 内网主机 -> 192.168.31.1（br-lan） -> 192.168.10.3(netmaker/yintan-gw) -> 192.168.10.1(netmaker/aliyun-gw) -> 172.19.227.25 这条链路会丢包
mysql
1、从抓包文件看，三次握手协商时mysql端（172.19.227.25）的Mss较小为1410；
2、从26号（seq=2932，len=454）报文和28号（seq=4784，len=7）报文的seq值可以看出，在26号和28号之间有一个seq=3386（2932+454），len=1398（4786-3386）的报文丢失；
3、从29号Dup ACK报文（seq=360，ack=3386）的ack值进一步确认客户端（192.168.31.118）没有收到mysql端（172.19.227.25）发送的seq=3386，len=1398的报文；
从mysql容器发送数据到客户端，数据包需要经过两层隧道封装，第一层是cilium网络插件，第二层是openwrt，在网络上传输还需要加上tcp和ip的头部信息，默认tcp头和ip头部的长度都是20，所以上述丢失的报文从网卡发出后的经过层层封装到达openwrt时长度已经超过mtu，1398+20(tcp头)+20(ip头)+40(openwrt隧道封装)+50(cilium封装)=1528，所以被丢弃。