Hadoop作业中的queue卡在75%

yarn-site.xml 配置:

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop0</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
    </property>
    <property>
       <name>yarn.scheduler.minimum-allocation-mb</name>
       <value>256</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>2048</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>1.0</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>2</value>
    </property>
    <property>
        <name>yarn.application.classpath</name>
        <value>$HADOOP_HOME/etc/hadoop,$HADOOP_HOME/share/hadoop/common/lib/*,$HADOOP_HOME/share/hadoop/common/*,$HADOOP_HOME/share/hadoop/hdfs,$HADOOP_HOME/share/hadoop/hdfs/lib/*,$HADOOP_HOME/share/hadoop/hdfs/*,$HADOOP_HOME/share/hadoop/mapreduce/*,$HADOOP_HOME/share/hadoop/yarn/lib/*,$HADOOP_HOME/share/hadoop/yarn/*</value>
    </property>
</configuration>

hadoop 跑在虚拟机上,配置如下:
CentOS7 4G内存,2核CPU,Swap2.5G(自己调整过)

直接跑 hadoop 自带的 wordcount 例子,在8088端口中监控作业运行进度,看到状态是 RUNNING,但卡在这个位置。

clipboard.png

翻开作业的syslog日志看这句话出现了很多次,一直卡在这里没法完成:

2018-02-04 11:40:38,240 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps

请问这是什么情况?大家有什么办法可以解决吗?

阅读 3.5k
撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进