我在使用Ambari 2.6, 安装的是HDP-2.6.4,使用Ranger管理hadoop的文件权限,并且给了用户yarn, admin在要用到的文件目录的所有权限。在使用这个系统的Oozie提交一个sqoop的任务时,发现执行sqoop的是用户是yarn,而不是提交任务的用户admin。
导致在执行的时候总是出现如下的错误:
28663 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - Failed with exception org.apache.hadoop.security.AccessControlException: Permission denied. user=yarn is not the owner of inode=part-m-00000.gz
28663 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - Failed with exception org.apache.hadoop.security.AccessControlException: Permission denied. user=yarn is not the owner of inode=part-m-00000.gz
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:285)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkOwner(FSPermissionChecker.java:285)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:260)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:260)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:428)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:428)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:365)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:365)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
28664 [Thread-28] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
我试过在集群的机器中执行统一的sqoop命令,没有任何问题。但是在oozie执行就有问题。
另外我查看Ranger的audit日志,发现了如下几条:
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.899","access":"READ","resource":"/tmp/test_tmp_ods/test_table","resType":"path","action":"read","result":1,"policy":8,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577377","seq_num":888721,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.900","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00001.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577378","seq_num":888723,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.900","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00000.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577379","seq_num":888725,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.901","access":"WRITE","resource":"/tmp/test_tmp_ods/test_table/part-m-00003.gz","resType":"path","action":"write","result":1,"policy":8,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577380","seq_num":888727,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.901","access":"WRITE","resource":"/tmp/test_tmp_ods/test_table/part-m-00003.gz","resType":"path","action":"write","result":1,"policy":8,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"ranger-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577381","seq_num":888729,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.901","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00002.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577382","seq_num":888731,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
{"repoType":1,"repo":"test_hadoop","reqUser":"yarn","evtTime":"2018-03-08 16:01:17.904","access":"EXECUTE","resource":"/tmp/test_tmp_ods/test_table/part-m-00003.gz","resType":"path","action":"execute","result":0,"policy":-1,"reason":"/tmp/test_tmp_ods/test_table","enforcer":"hadoop-acl","cliIP":"10.0.30.2","agentHost":"master","logType":"RangerAudit","id":"727c87c5-eeba-465e-ad8d-f1129c01801f-577383","seq_num":888733,"event_count":1,"event_dur_ms":0,"tags":[],"cluster_name":"test"}
也就是说yarn有读和写的权限,但是就是没有execute权限,导致了任务失败
有几个疑问:
- 为什么任务是用yarn这个用户执行的,而不是我提交任务的admin用户?我怎么才能让oozie中的sqoop action使用admin用户执行呢?
- 我非常确定在ranger中给了yarn用户所有权限,为什么ranger没有发现呢?
这是我在hortonworks提交的问题,也没有解决:
https://community.hortonworks...