为什么Oozie执行的Job使用yarn来执行而不是提交的用户

Question

为什么Oozie执行的Job使用yarn来执行而不是提交的用户

发布于
2018-03-08

更新于
2018-03-08

我在使用Ambari 2.6，安装的是HDP-2.6.4，使用Ranger管理hadoop的文件权限，并且给了用户yarn, admin在要用到的文件目录的所有权限。在使用这个系统的Oozie提交一个sqoop的任务时，发现执行sqoop的是用户是yarn，而不是提交任务的用户admin。

导致在执行的时候总是出现如下的错误：

28663 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  - Failed with exception org.apache.hadoop.security.AccessControlException: Permission denied. user=yarn is not the owner of inode=part-m-00000.gz
28663 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  - Failed with exception org.apache.hadoop.security.AccessControlException: Permission denied. user=yarn is not the owner of inode=part-m-00000.gz
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:285)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:285)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:260)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:260)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:428)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:428)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:365)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:365)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)

我试过在集群的机器中执行统一的sqoop命令，没有任何问题。但是在oozie执行就有问题。
另外我查看Ranger的audit日志，发现了如下几条：

{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.899","access":"READ","resource":"/tmp/test_tmp_ods/test_table","resType":"path","action":"read","result":1,"policy":8,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577377","seq_num":888721,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.900","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00001.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577378","seq_num":888723,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.900","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00000.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577379","seq_num":888725,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.901","access":"WRITE","resource":"/tmp/test_tmp_ods/test_table/part-m-00003.gz","resType":"path","action":"write","result":1,"policy":8,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577380","seq_num":888727,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.901","access":"WRITE","resource":"/tmp/test_tmp_ods/test_table/part-m-00003.gz","resType":"path","action":"write","result":1,"policy":8,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577381","seq_num":888729,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.901","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00002.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577382","seq_num":888731,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.904","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00003.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577383","seq_num":888733,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}

也就是说yarn有读和写的权限，但是就是没有execute权限，导致了任务失败

有几个疑问：

为什么任务是用yarn这个用户执行的，而不是我提交任务的admin用户？我怎么才能让oozie中的sqoop action使用admin用户执行呢？
我非常确定在ranger中给了yarn用户所有权限，为什么ranger没有发现呢？

这是我在hortonworks提交的问题，也没有解决：
https://community.hortonworks...

hadoop oozie

阅读 4.4k

撰写回答

你尚未登录，登录后可以

和开发者交流问题的细节
关注并接收问题和回答的更新提醒
参与内容的编辑和改进，让解决方法与时俱进

推荐问题

hive需要在配置文件中指定hadoop集群namenode的ip和端口吗，如果不需要，那他们是如何建立连接的呢？
hive需要在配置文件中指定hadoop集群namenode的ip和端口吗，如果不需要，那他们是如何建立连接的呢？hadoop：3.3.2hive：3.1.3
763 阅读

相似问题

找不到问题？创建新问题

为什么Oozie执行的Job使用yarn来执行而不是提交的用户

你尚未登录，登录后可以

hive需要在配置文件中指定hadoop集群namenode的ip和端口吗，如果不需要，那他们是如何建立连接的呢？