说说 HDFS fsck

1、描述

hdfs fsck 是 Hadoop 分布式文件系统 (HDFS) 的一个命令行工具，用于检查文件系统的一致性。它可以帮助管理员检查文件系统的健康状况，并提供有关丢失块、损坏文件等信息

2、基本用法

2.1、检查整个文件系统的一致性

hdfs fsck /

这将检查 HDFS 根目录下的所有文件和目录的一致性

2.2、检查特定目录或文件

hdfs fsck /path/to/directory_or_file

这将检查指定目录或文件的一致性

3、常用选项

3.1、-files

-files：显示文件名

hdfs fsck / -files

3.2、-blocks

-blocks：显示文件的块信息

hdfs fsck / -blocks

3.3、-locations

-locations：显示块的位置

hdfs fsck / -locations

3.4、-racks

-racks：显示块所在的机架信息

hdfs fsck / -racks

3.5、-openforwrite

-openforwrite：显示当前打开进行写入的文件

hdfs fsck / -openforwrite

3.6、-move

-move：尝试移动无法读取的块

hdfs fsck / -move

3.7、-delete

-delete：删除不可恢复的损坏文件

hdfs fsck / -delete

3.8、-list-corruptfileblocks

-list-corruptfileblocks：列出所有损坏的文件块。

hdfs fsck / -list-corruptfileblocks

4、示例

下面是一个更详细的命令示例，它检查 /dolphinscheduler/root/resources/mysql.sql 目录的文件系统一致性，并显示文件、块及其位置

[root@node subdir27]# hdfs fsck /dolphinscheduler/root/resources/mysql.sql  -files -blocks -locations
Connecting to namenode via http://10.253.26.86:9870/fsck?ugi=root&files=1&blocks=1&locations=1&path=%2Fdolphinscheduler%2Froot%2Fresources%2Fmysql.sql
FSCK started by root (auth:SIMPLE) from /10.253.26.86 for path /dolphinscheduler/root/resources/mysql.sql at Sat Aug 03 22:26:38 CST 2024

/dolphinscheduler/root/resources/mysql.sql 21 bytes, replicated: replication=3, 1 block(s):  Under replicated BP-1874052210-10.253.26.86-1694676061822:blk_1073940755_199940. Target Replicas is 3 but found 1 live replica(s), 0 decommissioned replica(s), 0 decommissioning replica(s).
0. BP-1874052210-10.253.26.86-1694676061822:blk_1073940755_199940 len=21 Live_repl=1  [DatanodeInfoWithStorage[10.253.26.86:9866,DS-a62ad891-19b4-46f0-9609-a79ba72cb13e,DISK]]


Status: HEALTHY
 Number of data-nodes:  1
 Number of racks:               1
 Total dirs:                    0
 Total symlinks:                0

Replicated Blocks:
 Total size:    21 B
 Total files:   1
 Total blocks (validated):      1 (avg. block size 21 B)
 Minimally replicated blocks:   1 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       1 (100.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    1
 Average block replication:     1.0
 Missing blocks:                0
 Corrupt blocks:                0
 Missing replicas:              2 (66.666664 %)
 Blocks queued for replication: 0

Erasure Coded Block Groups:
 Total size:    0 B
 Total files:   0
 Total block groups (validated):        0
 Minimally erasure-coded block groups:  0
 Over-erasure-coded block groups:       0
 Under-erasure-coded block groups:      0
 Unsatisfactory placement block groups: 0
 Average block group size:      0.0
 Missing block groups:          0
 Corrupt block groups:          0
 Missing internal blocks:       0
 Blocks queued for replication: 0
FSCK ended at Sat Aug 03 22:26:38 CST 2024 in 1 milliseconds


The filesystem under path '/dolphinscheduler/root/resources/mysql.sql' is HEALTHY

在上述输出中：

• CORRUPT 表示文件损坏。
• Total size 是文件的总大小。
• Total dirs 是目录的总数。
• Total files 是文件的总数。
• UNDER MIN REPL'D BLOCKS 表示未达到最小复制数的块数。
• Corrupt blocks 是损坏的块数。
• Missing replicas 是丢失的副本数