nfs客户端进程变D,延伸linux的lock

问题现象

客户端是opentack，通过nfs挂载我们存储，发现有个服务进程变D了，长时间无法恢复，如下

[root@ECM-043 ~]# ps aux|grep nova-compute
nova     21409  1.4  0.0 2117692 110916 ?      Dl   15:06   1:24 /opt/server/python27/bin/python /usr/bin/nova-compute --logfile /var/log/nova/compute.log

排查步骤

客户端排查，查找该进程hang 在哪里，lsof 查找进程打开的文件

lsof -p 21409 
nova-comp 21409 nova   22w   REG               0,29        0 2199054015793 /var/lib/nova/instances/locks/nova-storage-registry-lock (block.beijing.wocloud.cn:/var/share/ezfs/shareroot/block-bj)
可以看到客户端的进程打开了nfs上的nova-storage-registry-lock文件，下一步在服务端看下该文件的状态

服务端查看nova-storage-registry-lock 文件的inode号

知道了inode号

在所有的nfs服务端，看下是否有其他进程 拿着 该lock 文件

root@Storage5:~# cat /proc/locks |grep inode
1: POSIX  ADVISORY  WRITE 28331 00:14:2199053494291 0 EOF
2: POSIX  ADVISORY  WRITE 28165 00:14:1099511627783 0 0
3: POSIX  ADVISORY  READ  360769 00:0f:21590 4 4
4: POSIX  ADVISORY  WRITE 28331 00:14:2199054015793 0 EOF
5: POSIX  ADVISORY  WRITE 1022761 00:0f:673303453 0 EOF
6: POSIX  ADVISORY  WRITE 98842 00:0f:47181 0 EOF
7: POSIX  ADVISORY  WRITE 688677 00:0f:1186403778 0 EOF
8: POSIX  ADVISORY  READ  38754 00:0f:21590 4 4

服务端 确认哪些 nfs客户端连接

nfs 是无状态协议，和cifs不一样，不是TCP 吗？

root@Storage5:/var/lib/nfs/sm# ll
total 48
drwxr-xr-x 2 statd root 4096 Apr 13 09:03 ./
drwxr-xr-x 5 statd root 4096 Apr 10 18:21 ../
-rw------- 1 statd root   88 Apr 11 16:08 10.55.4.1
-rw------- 1 statd root   89 Apr 10 14:47 10.55.4.15
-rw------- 1 statd root   89 Apr 10 14:41 10.55.4.16
-rw------- 1 statd root   90 Apr  7 02:10 10.55.4.199
-rw-r----- 1 statd root 1350 Apr 11 13:42 10.55.4.205
-rw------- 1 statd root   89 Apr 11 16:00 10.55.4.31
-rw------- 1 statd root   89 Apr 10 14:44 10.55.4.32
-rw------- 1 statd root   89 Apr 10 14:34 10.55.4.37
-rw------- 1 statd root   89 Apr 11 16:04 10.55.4.54
-rw-r----- 1 statd root 2288 Apr 13 09:03 10.55.4.9
root@Storage5:/var/lib/nfs/sm# pwd
/var/lib/nfs/sm

延伸

linux下lock类型

文件锁，主要分为 flock 和fcntl

2者的粒度不一样

flock

锁住整个文件，如下，

root@scal61:/usr/share/pyshared/ezs3# cat /proc/locks 
1: FLOCK  ADVISORY  WRITE 97862 00:12:229635 0 EOF
2: FLOCK(lock类型）  ADVISORY（建议锁，非强制执行）  WRITE（holder能写lock文件） 4246（holder的pid） 00:12:20704（锁文件的MAJOR-DEVICE:MINOR-DEVICE:INODE-NUMBER） 0 EOF（锁文件的范围，0到EOF代表整个文件）
                 

fcntl

lock文件的某一部分，如下 posix

root@scal61:/usr/share/pyshared/ezs3# cat /proc/locks 
3: POSIX  ADVISORY  WRITE 97830 08:03:524404 0 EOF
4: POSIX  ADVISORY  WRITE 4056 00:12:41028 0 EOF
5: POSIX  ADVISORY  READ  2351 00:12:10095 4 4
6: POSIX  ADVISORY  READ  2783 00:12:10052 4 4
7: POSIX  ADVISORY  READ  2783 00:12:14432 4 4
8: POSIX  ADVISORY  WRITE 2783 00:12:10050 0 0

---

* centos

• FLOCK signifying the older-style UNIX file locks from a flock system
  call
• POSIX representing the newer POSIX locks from the lockf system call.

• ADVISORY means that the lock does not prevent other people from accessing the data; it only prevents other attempts to lock it
• MANDATORY means that no other access to the data is permitted while the lock is held
• The fourth column reveals whether the lock is allowing the holder READ or WRITE access to the file