新手搭建hadoop集群深坑

DataNode不启动

搭建hadoop集群可以参考我之前的几篇文章.
我的hadoop集群版本是3.3.3
此处记录一下困扰了好几天的问题:
我的hadoop集群如下
hadoop-4
hadoop-5
hadoop-6
部署完成后,使用start-dfs.sh启动.有个问题.
假如我在hadoop-4启动,集群中所有组件都能正常启动,但是datanode只有hadoop-4正常启动.其他的节点不启动datanode.也没有任何的报错信息.
如果你遇到了和我一样的问题.
你只需要在hadoop-3.3.3/etc/hadoop下找到works文件,将其中的localhost改成你的所有集群节点

hadoop-4
hadoop-5
hadoop-6

就这样重启即可.

注：
解决方案参考下文:https://blog.csdn.net/qq_3833...

HA-Hadoop集群active节点不自动切换

zkfc的log报错如下

2022-10-30 16:08:25,892 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Authentications that can continue: publickey,keyboard-interactive,password
2022-10-30 16:08:25,892 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Next authentication method: publickey
2022-10-30 16:08:25,892 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Authentications that can continue: password
2022-10-30 16:08:25,892 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Next authentication method: password
2022-10-30 16:08:25,892 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Disconnecting from hadoop-6 port 22
2022-10-30 16:08:25,892 WARN org.apache.hadoop.ha.SshFenceByTcpPort: Unable to connect to hadoop-6 as user root
com.jcraft.jsch.JSchException: Auth fail
    at com.jcraft.jsch.Session.connect(Session.java:519)
    at org.apache.hadoop.ha.SshFenceByTcpPort.tryFence(SshFenceByTcpPort.java:99)
    at org.apache.hadoop.ha.NodeFencer.fence(NodeFencer.java:113)
    at org.apache.hadoop.ha.NodeFencer.fence(NodeFencer.java:92)
    at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:557)
    at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:530)
    at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:63)
    at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:966)
    at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:997)
    at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
    at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:476)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:636)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2022-10-30 16:08:25,892 WARN org.apache.hadoop.ha.NodeFencer: Fencing method org.apache.hadoop.ha.SshFenceByTcpPort(null) was unsuccessful.
2022-10-30 16:08:25,892 ERROR org.apache.hadoop.ha.NodeFencer: Unable to fence service by any configured method.
2022-10-30 16:08:25,893 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
java.lang.RuntimeException: Unable to fence NameNode at hadoop-6/192.168.0.120:8020
    at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:558)
    at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:530)
    at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:63)
    at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:966)
    at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:997)
    at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
    at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:476)
    at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:636)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)

添加shell的这段配置

<property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence
               shell(/bin/true)
        </value>
    </property>

解决参考如下:http://www.wjhsh.net/liuys635...