istio部署在virtual machine 与k8s 通讯问题

cfanbo
  • 658

部署需求:
肉网将 istio 安装在 vm 上,实现网格服务,共三台机器。网络插件是 Flannel

master: 192.168.3.58
vm1: 192.168.3.120
vm2: 192.168.3.56

参考官方教程:https://istio.io/latest/docs/...

安装k8s集群工具 kubeadm,将安装了负载均衡器 metallb,使用的 Layer 2 模式(教程:https://mp.weixin.qq.com/s/Z4...

# k8s-master
root@sxf-virtual-machine:/home/sxf# ksvc -A
NAMESPACE      NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                                                           AGE
default        kubernetes              ClusterIP      10.96.0.1        <none>          443/TCP                                                           168m
istio-system   istio-eastwestgateway   LoadBalancer   10.106.10.60     192.168.3.251   15021:32430/TCP,15443:30670/TCP,15012:31969/TCP,15017:30699/TCP   135m
istio-system   istio-ingressgateway    LoadBalancer   10.106.3.144     192.168.3.252   15021:30271/TCP,80:31320/TCP,443:30527/TCP                        157m
istio-system   istiod                  ClusterIP      10.101.237.7     <none>          15010/TCP,15012/TCP,443/TCP,15014/TCP                             162m
kube-system    kube-dns                ClusterIP      10.96.0.10       <none>          53/UDP,53/TCP,9153/TCP                                            168m

# 查看代理信息
root@sxf-virtual-machine:/home/sxf# istioctl proxy-status
NAME                                                    CDS        LDS        EDS        RDS          ISTIOD                      VERSION
helloworld-v1-776f57d5f6-9pxn7.sample                   SYNCED     SYNCED     SYNCED     SYNCED       istiod-5bdf585695-l2845     1.10.3
helloworld-v2-54df5f84b-gfqtz.sample                    SYNCED     SYNCED     SYNCED     SYNCED       istiod-5bdf585695-l2845     1.10.3
istio-eastwestgateway-6fd76487d6-2hndb.istio-system     SYNCED     SYNCED     SYNCED     NOT SENT     istiod-5bdf585695-l2845     1.10.3
istio-ingressgateway-697dc5b889-b54nn.istio-system      SYNCED     SYNCED     SYNCED     NOT SENT     istiod-5bdf585695-l2845     1.10.3
vm1.vm                                                  SYNCED     SYNCED     SYNCED     SYNCED       istiod-5bdf585695-l2845     1.10.0
vm2.vm                                                  SYNCED     SYNCED     SYNCED     SYNCED       istiod-5bdf585695-l2845     1.10.0

# 在master宿主机通过ip可以访问helloworld服务,但域名不可以,提示无法解析(这里应该是正常的,只有pod之间才可以通过域名访问)
root@sxf-virtual-machine:/home/sxf# curl  10.111.123.197:5000/hello
Hello version: v2, instance: helloworld-v2-54df5f84b-gfqtz

root@sxf-virtual-machine:/home/sxf# curl helloworld.sample.svc:5000/hello
curl: (6) Could not resolve host: helloworld.sample.svc

网关服务 istio-eastwestgateway 的LB 地址为 192.168.3.251。目前可以看到两台vm的信息。

root@sxf-virtual-machine:/home/sxf# k get wg  -A
NAMESPACE   NAME    AGE
vm          myapp   133m

root@sxf-virtual-machine:/home/sxf# k get we  -A
NAMESPACE   NAME                  AGE    ADDRESS
vm          myapp-192.168.3.120   111m   192.168.3.120
vm          myapp-192.168.3.56    94m    192.168.3.56

在教程中的 "验证安装istio是否成功" 步骤中,istio.log日志正常。但服务 curl helloworld.sampel.svc 无法访问,但可以通过dns查找到服务对应的ip地址 10.111.123.197

# vm1 无法访问
root@vm1:/home/sxf# curl helloworld.sample.svc:5000/hello
upstream connect error or disconnect/reset before headers. reset reason: connection failure


# 解析到的信息,ip是正确的
root@vm1:/home/sxf# nslookup helloworld.sample.svc
Server:         127.0.0.53
Address:        127.0.0.53#53

Name:   helloworld.sample.svc
Address: 10.111.123.197

# 192.168.3.1是网关地址, 3.2是公司内部的BGP网关
root@vm1:/home/sxf# ping helloworld.sample.svc
PING helloworld.sample.svc (10.111.123.197) 56(84) bytes of data.
From _gateway (192.168.3.1): icmp_seq=3 Redirect Network(New nexthop: 192.168.3.2 (192.168.3.2))

# 端口是通的
root@vm1:/home/sxf# telnet helloworld.sample.svc 5000
Trying 10.111.123.197...
Connected to helloworld.sample.svc.
Escape character is '^]'.

以下是在vm1上的istio域名解析日志

2021-08-04T08:39:41.968114Z     debug   dns     request ;; opcode: QUERY, status: NOERROR, id: 61474
;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;helloworld.sample.svc. IN       AAAA

;; ADDITIONAL SECTION:

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: ; udp: 1200
        protocol=udp edns=true id=ae0dc6d2-d888-473e-b873-136263c1d714
2021-08-04T08:39:41.968371Z     debug   dns     response for hostname "helloworld.sample.svc." (found=true): ;; opcode: QUERY, status: NOERROR, id: 61474
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;helloworld.sample.svc. IN       AAAA
        protocol=udp edns=true id=ae0dc6d2-d888-473e-b873-136263c1d714
2021-08-04T08:39:41.968115Z     debug   dns     request ;; opcode: QUERY, status: NOERROR, id: 48365
;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;helloworld.sample.svc. IN       A

;; ADDITIONAL SECTION:

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: ; udp: 1200
        protocol=udp edns=true id=e89119b5-37b5-4434-aad1-b3b0dfbf15f0
2021-08-04T08:39:41.968786Z     debug   dns     response for hostname "helloworld.sample.svc." (found=true): ;; opcode: QUERY, status: NOERROR, id: 48365
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;helloworld.sample.svc. IN       A

;; ANSWER SECTION:
helloworld.sample.svc.  30      IN      A       10.111.123.197
        protocol=udp edns=true id=e89119b5-37b5-4434-aad1-b3b0dfbf15f0

可以看到几个关键点 found=truehelloworld.sample.svc. 30 IN A 10.111.123.197, 所以这里dns 也应该是没有问题的。

master 主机信息

root@sxf-virtual-machine:/home/sxf# ip route
default via 192.168.3.1 dev ens160 proto dhcp metric 100 
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1 
169.254.0.0/16 dev ens160 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.3.0/24 dev ens160 proto kernel scope link src 192.168.3.58 metric 100 

# hosts
root@sxf-virtual-machine:/home/sxf# more /etc/hosts
127.0.0.1       localhost
127.0.1.1       sxf-virtual-machine

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

# nameserver
root@sxf-virtual-machine:/home/sxf# more /etc/resolv.conf
nameserver 127.0.0.53
options edns0
search anee.com.cn

vm1主机信息

# hosts
root@vm1:/home/sxf# more /etc/hosts
127.0.0.1       localhost
127.0.1.1       vm1

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.3.251 istiod.istio-system.svc

# nameserver
root@vm1:/home/sxf# more /etc/resolv.conf
nameserver 127.0.0.53
options edns0

# 路由信息
root@vm1:/home/sxf# ip route
default via 192.168.3.1 dev ens36 proto static metric 20100 
169.254.0.0/16 dev ens36 scope link metric 1000 
192.168.3.0/24 dev ens36 proto kernel scope link src 192.168.3.120 metric 100 

root@vm1:/home/sxf# route -n
内核 IP 路由表
目标            网关            子网掩码        标志  跃点   引用  使用 接口
0.0.0.0         192.168.3.1     0.0.0.0         UG    20100  0        0 ens36
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 ens36
192.168.3.0     0.0.0.0         255.255.255.0   U     100    0        0 ens36

# 网卡信息
root@vm1:/home/sxf# ifconfig
ens36: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.3.120  netmask 255.255.255.0  broadcast 192.168.3.255
        inet6 fe80::8e95:5567:8fa4:e0e7  prefixlen 64  scopeid 0x20<link>
        ether 00:0c:29:e7:86:b1  txqueuelen 1000  (以太网)
        RX packets 2164253  bytes 206952534 (206.9 MB)
        RX errors 0  dropped 24275  overruns 0  frame 0
        TX packets 112319  bytes 18330399 (18.3 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (本地环回)
        RX packets 26287  bytes 13980478 (13.9 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 26287  bytes 13980478 (13.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        
        

# iptables
root@vm1:/home/sxf# iptables-save
# Generated by iptables-save v1.6.1 on Wed Aug  4 17:21:36 2021
*filter
:INPUT ACCEPT [29263:14149342]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [14082:6893008]
COMMIT
# Completed on Wed Aug  4 17:21:36 2021
# Generated by iptables-save v1.6.1 on Wed Aug  4 17:21:36 2021
*mangle
:PREROUTING ACCEPT [313262:101209145]
:INPUT ACCEPT [308840:100700735]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [101662:27585942]
:POSTROUTING ACCEPT [101762:27615155]
COMMIT
# Completed on Wed Aug  4 17:21:36 2021
# Generated by iptables-save v1.6.1 on Wed Aug  4 17:21:36 2021
*nat
:PREROUTING ACCEPT [3954:385133]
:INPUT ACCEPT [3911:377509]
:OUTPUT ACCEPT [536:37403]
:POSTROUTING ACCEPT [657:46151]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -d 192.168.3.58/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A OUTPUT -p udp -m udp --dport 53 -m owner --uid-owner 123 -j RETURN
-A OUTPUT -p udp -m udp --dport 53 -m owner --gid-owner 123 -j RETURN
-A OUTPUT -d 127.0.0.53/32 -p udp -m udp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -p tcp -m tcp ! --dport 53 -m owner --uid-owner 123 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp -m tcp ! --dport 53 -m owner ! --uid-owner 123 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 123 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 123 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -p tcp -m tcp ! --dport 53 -m owner ! --gid-owner 123 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 123 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.53/32 -p tcp -m tcp --dport 53 -j REDIRECT --to-ports 15053
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Wed Aug  4 17:21:36 2021
回复
阅读 1.2k
2 个回答
✓ 已被采纳

手动在vm1 上添加了一条路由规则可以了,但按文档介绍会自动更新这个规则的,并不需要手动添加。

route add -net 10.244.0.0 gw 192.168.3.58 netmask 255.255.0.0

将网络段 10.244.0.0/16 的所有请求转到 192.168.3.58 ip
这个网段是创建k8s是指定的pod的cird

看起来像是网络问题,在master和vm上trace一下路由是否有什么不一样的地方?

撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进
宣传栏