Hive运行时MapReduce、NodeManager内存不足解决办法

Hive任务运行时报错：
Status: Failed------------------------------------------------------------------
Application application_1574770014204_0003 failed 2 times due to AM Container for appattempt_1574770014204_0003_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://hadoop103:8088/cluster/app/application_1574770014204_0003Then, click on links to logs of each attempt.
Diagnostics: Container [pid=5400,containerID=container_1574770014204_0003_02_000001] is running beyond virtual memory limits. Current usage: 293.5 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1574770014204_0003_02_000001 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 5400 5398 5400 5400 (bash) 1 0 108658688 297 /bin/bash -c /opt/module/jdk1.8.0_144/bin/java  -Xmx819m -Djava.io.tmpdir=/opt/module/hadoop-2.7.2/data/tmp/nm-local-dir/usercache/root/appcache/application_1574770014204_0003/container_1574770014204_0003_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-2.7.2/logs/userlogs/application_1574770014204_0003/container_1574770014204_0003_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/opt/module/hadoop-2.7.2/logs/userlogs/application_1574770014204_0003/container_1574770014204_0003_02_000001/stdout 2>/opt/module/hadoop-2.7.2/logs/userlogs/application_1574770014204_0003/container_1574770014204_0003_02_000001/stderr  
|- 5447 5400 5400 5400 (java) 663 143 2782834688 74838 /opt/module/jdk1.8.0_144/bin/java -Xmx819m -Djava.io.tmpdir=/opt/module/hadoop-2.7.2/data/tmp/nm-local-dir/usercache/root/appcache/application_1574770014204_0003/container_1574770014204_0003_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-2.7.2/logs/userlogs/application_1574770014204_0003/container_1574770014204_0003_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel= org.apache.tez.dag.app.DAGAppMaster --session 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Application application_1574770014204_0003 failed 2 times due to AM Container for appattempt_1574770014204_0003_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://hadoop103:8088/cluster/app/application_1574770014204_0003Then, click on links to logs of each attempt.
Diagnostics: Container [pid=5400,containerID=container_1574770014204_0003_02_000001] is running beyond virtual memory limits. Current usage: 293.5 MB of 1 GB physical memory used; 2.7 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1574770014204_0003_02_000001 :

|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 5400 5398 5400 5400 (bash) 1 0 108658688 297 /bin/bash -c /opt/module/jdk1.8.0_144/bin/java  -Xmx819m -Djava.io.tmpdir=/opt/module/hadoop-2.7.2/data/tmp/nm-local-dir/usercache/root/appcache/application_1574770014204_0003/container_1574770014204_0003_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-2.7.2/logs/userlogs/application_1574770014204_0003/container_1574770014204_0003_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/opt/module/hadoop-2.7.2/logs/userlogs/application_1574770014204_0003/container_1574770014204_0003_02_000001/stdout 2>/opt/module/hadoop-2.7.2/logs/userlogs/application_1574770014204_0003/container_1574770014204_0003_02_000001/stderr  
|- 5447 5400 5400 5400 (java) 663 143 2782834688 74838 /opt/module/jdk1.8.0_144/bin/java -Xmx819m -Djava.io.tmpdir=/opt/module/hadoop-2.7.2/data/tmp/nm-local-dir/usercache/root/appcache/application_1574770014204_0003/container_1574770014204_0003_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-2.7.2/logs/userlogs/application_1574770014204_0003/container_1574770014204_0003_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel= org.apache.tez.dag.app.DAGAppMaster --session 

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.

解决办法如下：
在Hive的conf目录下的hive-site.xml配置文件中添加如下几个配置：

<property>
    <name>mapreduce.map.memory.mb</name>
    <value>2048</value>
    <description>设置mapTask的默认内存大小</description>
</property>
<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>2048</value>
    <description>设置reduceTask的默认内存大小</description>
</property>
<property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>2048</value>
    <description>设置mnodemanager的默认内存大小:2048*4.2</description>
</property>

或者在HQL执行前先执行如下命令：

//设置mapTask的内存大小
set mapreduce.map.memory.mb=2048;
//设置reduceTask的内存大小
set mapreduce.reduce.memory.mb=2048;
//设置mnodemanager的内存大小:2048*4.2
set yarn.nodemanager.vmem-pmem-ratio=4.2;