hive语句执行报错排查分析

原文链接：https://segmentfault.com/a/1190000043306816

Spark job[9] failed
java.util.concurrent.ExecutionException: Exception thrown by job
 
Error while processing statement: FAILED: Execution Error, return code 3 from 
org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed due to task 
failures: Job aborted due to stage failure: Task 1 in stage 23.0 failed 4 times
, most recent failure: Lost task 1.3 in stage 23.0 (TID 2221, hadoopslave, 
executor 18): UnknownReason Driver stacktrace:

拿到 application_1664415573734_40190
CDH-CM后台 Spark History Server 查找日志
Show incomplete applications

 Fatal error: org.apache.hadoop.hive.ql.metadata.HiveFatalException: 
[Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. 
The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions 
and hive.exec.max.dynamic.partitions.pernode. 
Maximum was set to 100 partitions per node, number of dynamic partitions on this node: 101  

原文链接：https://segmentfault.com/a/1190000043306816

解决方法：

hive.exec.max.dynamic.partitions.pernode
默认值：100

在每个执行MR的节点上，最大可以创建多少个动态分区。

该参数需要根据实际的数据来设定。

比如：源数据中包含了一年的数据，即day字段有365个值，那么该参数就需要设置成大于365，如果使用默认值100，则会报错。

SET hive.exec.dynamic.partition=true;
SET hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.max.dynamic.partitions.pernode = 1000;
SET hive.exec.max.dynamic.partitions=1000;
在所有执行MR的节点上，最大一共可以创建多少个动态分区

参考文章：
http://lxw1234.com/archives/2015/06/286.htm
https://blog.csdn.net/cclovezbf/article/details/126950236
https://www.jianshu.com/p/962fa4b4ca13