hadoop streaming 运行时 subprocess failed with code 126

在运行hadoop streaming时,报错,异常内容如下:

Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 126
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)

    hadoop的启动shell脚本如下:

hdfs_input="/test/testLine.log"
hdfs_ouput="/out"
python_lib="${fs_default_name}/python/python.tar.gz"

WORK_DIR=$(cd `dirname $0`; pwd)

${hadoop_path}/bin/hadoop jar ${hadoop_path}/libexec/share/hadoop/tools/lib/hadoop-streaming-2.8.2.jar \
    -archives ${python_lib}\#lib \
    -D mapreduce.job.name="didp_log" \
    -D mapreduce.job.reduces=1 \
    -input ${hdfs_input} \
    -output ${hdfs_ouput} \
    -mapper "lib/python/bin/python mapper.py" \
    -reducer "lib/python/bin/python reducer.py" \
    -file ${WORK_DIR}/mapper.py \
    -file ${WORK_DIR}/reducer.py \
    -inputformat org.apache.hadoop.mapred.TextInputFormat \
    -outputformat org.apache.hadoop.mapred.TextOutputFormat \

求大神告知如何处理? 多谢

阅读 3.1k
撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进
推荐问题