hadoop job运行时遇到错误:java.io.IOException: Permission denied

今天在一台新的机器上部署程序,运行job时,遇到这个错误:

Exception in thread "main" java.io.IOException: Permission denied
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.checkAndCreate(File.java:1704)
        at java.io.File.createTempFile(File.java:1792)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:146)

很明显是权限问题,查后发现是在org.apache.hadoop.util.RunJar.main 里面要创建临时文件:

    File tmpDir = new File(new Configuration().get("hadoop.tmp.dir"));
    ensureDirectory(tmpDir);

    final File workDir = File.createTempFile("hadoop-unjar", "", tmpDir);
    if (!workDir.delete()) {
      System.err.println("Delete failed for " + workDir);
      System.exit(-1);
    }

hadoop.tmp.dir这个目录存放的是解压后的jar文件等临时文件。

那么hadoop.tmp.dir这个变量是在哪里定义的呢?于是查看本机提交job时用到的hadoop配置文件core-site.xml:

<property>
  <name>user.home.name</name>
  <value>/home/hadoop</value>
  <description>
  </description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://hadoop001:9000</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>

<property>
  <name>hadoop.tmp.dir</name>
  <value>${user.home.name}/hadoop-data</value>
  <description>A base for other temporary directories.</description>
</property>

发现hadoop.tmp.dir是 /home/hadoop/hadoop-data 这个目录,而当前的admin用户确实是没有权限访问这个目录的。于是改下配置文件就可以了。

在google上查的时候发现已经有人提出“RunJar should provide more diags when it can’t create a temp file”,确实这个提示信息太不友好了。其中提到还有一种是jar包损坏也会导致这个错误,见:https://issues.apache.org/jira/browse/HADOOP-6614

发表评论

电子邮件地址不会被公开。 必填项已用*标注