为什么Oozie执行的Job使用yarn来执行而不是提交的用户

我在使用Ambari 2.6, 安装的是HDP-2.6.4,使用Ranger管理hadoop的文件权限,并且给了用户yarn, admin在要用到的文件目录的所有权限。在使用这个系统的Oozie提交一个sqoop的任务时,发现执行sqoop的是用户是yarn,而不是提交任务的用户admin。

导致在执行的时候总是出现如下的错误:

28663 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  - Failed with exception org.apache.hadoop.security.AccessControlException: Permission denied. user=yarn is not the owner of inode=part-m-00000.gz
28663 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  - Failed with exception org.apache.hadoop.security.AccessControlException: Permission denied. user=yarn is not the owner of inode=part-m-00000.gz
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:285)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:285)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:260)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:260)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:428)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:428)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:365)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:365)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
28664 [Thread-28] INFO  org.apache.sqoop.hive.HiveImport  -     at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)

我试过在集群的机器中执行统一的sqoop命令,没有任何问题。但是在oozie执行就有问题。
另外我查看Ranger的audit日志,发现了如下几条:

{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.899","access":"READ","resource":"/tmp/test_tmp_ods/test_table","resType":"path","action":"read","result":1,"policy":8,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577377","seq_num":888721,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.900","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00001.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577378","seq_num":888723,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.900","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00000.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577379","seq_num":888725,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.901","access":"WRITE","resource":"/tmp/test_tmp_ods/test_table/part-m-00003.gz","resType":"path","action":"write","result":1,"policy":8,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577380","seq_num":888727,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.901","access":"WRITE","resource":"/tmp/test_tmp_ods/test_table/part-m-00003.gz","resType":"path","action":"write","result":1,"policy":8,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577381","seq_num":888729,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.901","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00002.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577382","seq_num":888731,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.904","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00003.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577383","seq_num":888733,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}

也就是说yarn有读和写的权限,但是就是没有execute权限,导致了任务失败

有几个疑问:

  1. 为什么任务是用yarn这个用户执行的,而不是我提交任务的admin用户?我怎么才能让oozie中的sqoop action使用admin用户执行呢?
  2. 我非常确定在ranger中给了yarn用户所有权限,为什么ranger没有发现呢?

这是我在hortonworks提交的问题,也没有解决:
https://community.hortonworks...

阅读 4.4k
撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进