PNFS集群 client 10036(协议不对称) 是什么原因?

Question

pnfs 集群协议不对称问题

公司需要做pnfs相关研究, 按照 https://people.freebsd.org/~rmacklem/pnfs-planb-setup.txt 搭建了一个 pnfs 的集群, 挂载是按照nfs4.2 的方式挂的, 创建文件的时候报 Input/Output err, 但文件还是创建出来了, 日志报10036, 查了官方的错误码 (https://www.tech-invite.com/y75/tinv-ietf-rfc-7530-10.html#e-...) 是因为 rpc 协议不对称

操作

root@client:~ # mount_nfs -o nfsv4,minorversion=2,pnfs 192.168.0.13:/mds /mnt/client/
root@client:~ # df
Filesystem        1K-blocks    Used    Avail Capacity  Mounted on
/dev/da0s1a        19279772 2378388 15359004    13%    /
devfs                     1       0        1     0%    /dev
192.168.0.13:/mds  38559544 9391556 26083228    26%    /mnt/client


root@client:~ # touch /mnt/client/2
touch: /mnt/client/2: Input/output error

root@client:~ # ls /mnt/client/
2

客户端的日志:

Feb 28 00:14:42 client kernel: nfsv4 client/server protocol prob err=10036
Feb 28 00:15:21 client syslogd: last message repeated 2 times
Feb 28 00:16:38 client syslogd: last message repeated 1 times

mds日志:

Feb 28 18:52:49 mds kernel: op=22
Feb 28 18:52:49 mds kernel: op=3
Feb 28 18:52:49 mds kernel: op=9
Feb 28 18:52:49 mds kernel: nfsvno_getattr: gotattr=0 err=0 chg=18446744071577416809
Feb 28 18:52:49 mds kernel: op=53
Feb 28 18:52:49 mds kernel: op=22
Feb 28 18:52:49 mds kernel: op=9
Feb 28 18:52:49 mds kernel: nfsvno_getattr: gotattr=0 err=0 chg=18446744071577356621
Feb 28 18:52:49 mds kernel: op=15
Feb 28 18:52:49 mds kernel: op=53
Feb 28 18:52:49 mds kernel: op=22
Feb 28 18:52:49 mds kernel: op=9
Feb 28 18:52:49 mds kernel: nfsvno_getattr: gotattr=0 err=0 chg=18446744071577356621
Feb 28 18:52:49 mds kernel: op=15
Feb 28 18:52:49 mds kernel: op=53
Feb 28 18:52:49 mds kernel: op=22
Feb 28 18:52:49 mds kernel: op=18
Feb 28 18:52:49 mds kernel: nfsvno_getattr: gotattr=0 err=0 chg=1
Feb 28 18:52:49 mds kernel: nfsrv_pnfscreate: cruid=0 crgid=0
Feb 28 18:52:49 mds kernel: nfsrv_dscreate: dvp=0xfffff80025e63700 fname=02d9be67dba951eb0c000000ceaf2400b9bbfd3d0000000000000000
Feb 28 18:52:49 mds kernel: nfsrv_dscreate: setattr-uid=0
Feb 28 18:52:49 mds kernel: nfsrv_pnfscreate: mirrorcnt=1 maxmirror=1
Feb 28 18:52:49 mds kernel: in nfsrv_proxyds
Feb 28 18:52:49 mds kernel: nfsvno_getattr: gotattr=1 err=0 chg=4
Feb 28 18:52:49 mds kernel: nfsvno_getattr: gotattr=0 err=0 chg=18446735277675193600
Feb 28 18:52:49 mds kernel: op=32
Feb 28 18:52:49 mds kernel: op=10
Feb 28 18:52:49 mds kernel: op=9
Feb 28 18:52:49 mds kernel: in nfsrv_proxyds
Feb 28 18:52:49 mds kernel: nfsvno_getattr: gotattr=1 err=0 chg=4
Feb 28 18:52:49 mds kernel: op=22
Feb 28 18:52:49 mds kernel: op=9
Feb 28 18:52:49 mds kernel: nfsvno_getattr: gotattr=0 err=0 chg=18446744071577356621
Feb 28 18:52:49 mds kernel: op=31
Feb 28 18:52:49 mds kernel: op=50
Feb 28 18:52:49 mds kernel: layoutget ltyp=1 iom=2 off=0 len=18446744073709551615 mlen=0
Feb 28 18:52:49 mds kernel: layoutget getfh=0
Feb 28 18:52:49 mds kernel: layoutget findlay=-1
Feb 28 18:52:49 mds kernel: nfsrv_dsgetsockmnt: mirrorcnt=1 err=0
Feb 28 18:52:49 mds kernel: gotone=1
Feb 28 18:52:49 mds kernel: eo nfsrv_dsgetsockmnt: gotone=1 err=0
Feb 28 18:52:49 mds kernel: layoutget devandfh=0
Feb 28 18:52:49 mds kernel: layoutget addl=0
Feb 28 18:52:49 mds kernel: nfsrv_layoutget stat=0 layoutlen=72
Feb 28 18:52:55 mds kernel: op=53

ds 日志:

Feb 28 18:52:44 ds2 syslogd: last message repeated 7 times
Feb 28 18:52:44 ds2 kernel: op=22
Feb 28 18:52:44 ds2 kernel: op=9
Feb 28 18:52:44 ds2 kernel: nfsvno_getattr: gotattr=0 err=0 chg=18446744071577356621
Feb 28 18:52:44 ds2 kernel: op=53
Feb 28 18:52:44 ds2 kernel: op=22
Feb 28 18:52:44 ds2 kernel: op=18
Feb 28 18:52:44 ds2 kernel: nfsvno_getattr: gotattr=0 err=0 chg=1
Feb 28 18:52:44 ds2 kernel: nfsrv_pnfscreate: no srv
Feb 28 18:52:44 ds2 kernel: nfsvno_getattr: gotattr=0 err=0 chg=18446735277854846976
Feb 28 18:52:44 ds2 kernel: nfsvno_getattr: gotattr=0 err=0 chg=18446735278183855872
Feb 28 18:52:44 ds2 kernel: op=10
Feb 28 18:52:44 ds2 kernel: op=9
Feb 28 18:52:44 ds2 kernel: nfsvno_getattr: gotattr=0 err=0 chg=5
Feb 28 18:52:44 ds2 kernel: op=22
Feb 28 18:52:44 ds2 kernel: op=9
Feb 28 18:52:44 ds2 kernel: nfsvno_getattr: gotattr=0 err=0 chg=18446744071577356621
Feb 28 18:52:44 ds2 kernel: op=53
Feb 28 18:52:44 ds2 kernel: op=22
Feb 28 18:52:44 ds2 kernel: op=9
Feb 28 18:52:44 ds2 kernel: nfsvno_getattr: gotattr=0 err=0 chg=18446744071577356621
Feb 28 18:52:44 ds2 kernel: op=34
Feb 28 18:52:44 ds2 kernel: nfsvno_getattr: gotattr=0 err=0 chg=120259084288
Feb 28 18:52:44 ds2 kernel: in nfsrv_proxyds
Feb 28 18:52:44 ds2 syslogd: last message repeated 2 times
Feb 28 18:52:44 ds2 kernel: op=9
Feb 28 18:52:44 ds2 kernel: nfsvno_getattr: gotattr=0 err=0 chg=0
Feb 28 18:52:44 ds2 kernel: op=53
Feb 28 18:52:44 ds2 kernel: op=22
Feb 28 18:52:44 ds2 kernel: op=4
Feb 28 18:52:44 ds2 kernel: in nfsrv_proxyds

mds /etc/exports配置:

V4: / -sec=sys -network 192.168.0.0 -mask 255.255.255.0
/mds -sec=sys -maproot=root -network 192.168.0.0 -mask 255.255.255.0

mds /etc/fstab配置:

root@mds:~ # cat /etc/fstab
# Device        Mountpoint      FStype  Options Dump    Pass#
/dev/da0s1a     /               ufs     rw      1       1
/dev/da0s1b     none            swap    sw      0       0
192.168.0.11:/  /ds1            nfs     rw,nfsv4,minorversion=2,soft,retrans=2 0 0
192.168.0.12:/  /ds2            nfs     rw,nfsv4,minorversion=2,soft,retrans=2 0 0

ds 配置:

root@ds2:~ # cat /etc/exports
V4: /ds2
/ds2 -sec=sys -maproot=root 192.168.0.13
/ds2 -sec=sys 192.168.0.10 192.168.0.8

root@ds1:~ # cat /etc/exports
V4: /ds1
/ds1 -sec=sys -maproot=root 192.168.0.13
/ds1 -sec=sys 192.168.0.10 192.168.0.8

初步分析挂载是4.2的, 使用4.2协议与服务端通信后不成功又用了nfs3 的协议重新创建文件, 所以文件还是创建出来了, rpc 协议不对称这种情况是否要重新编译nfs内核rpc模块

或者有其他方式搭建 pnfs 集群的