2013-11-29 68 views
0

我一直在玩Hadoop和它的姊妹项目,我一直有一些问题,但我终于击中了一个,我不能找到答案:Hive MapReduce作业提交失败“目标是一个目录”

我有一个hive表存储在hdfs作为制表符分隔的文本文件。我可以做餐桌上的基本选择,但是当我使查询稍微复杂一点,蜂巢把它变成一个映射精简工作,这失败与以下堆栈跟踪

13/11/29 08:31:00 ERROR security.UserGroupInformation:PriviledgedActionException as:hduser(auth:SIMPLE)cause:java.io.IOException:Target/tmp/hadoop->> yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib是一个目录 13/11/29 08:31:00错误security.UserGroupInformation:PriviledgedActionException as:hduser(auth:SIMPLE)cause:java.io.IOException:Target/tmp/hadoop-yarn/staging/hduser /。 staging/job_1385633903169_0013/libjars/lib/lib是一个目录 java.io.IOException:Target /tmp/hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib是目录 ,位于org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:500) at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:502) at org.apache .hadoop.fs.FileUtil.copy(FileUtil.java:348) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) at org.apache.hadoop.fs.FileUtil.copy(FileUtil的.java:289) 在org.apache.hadoop.mapreduce.JobSubmitter.copyRemoteFiles(JobSubmitter.java:139) 在org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:212) 在org.apache .hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387) 在org.apache.hadoop.mapreduce.Job $ 10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job $ 10.run(Job.java:1265) at java.security.AccessController.doPrivileged (Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop。 (org.apache.hadoop.mapred.JobClient)$ 1.run(JobClient.java:562) at org.apache.hadoop.mapred.JobClient $ 1.run(JobClient。 java:557) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupI nformation.doAs(UserGroupInformation.java:1491) 在org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557) 在org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:425) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java: 144) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) at org.apache。 hadoop.hive.ql.Driver.runInternal(Driver.java:1020) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache。 hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli。 CliDriver.executeDriver(clidriver中。java:781) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) at sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法) 在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 在java.lang.reflect中.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) 作业提交失败,异常为'java.io.IOException(Target/tmp/hadoop- yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib是一个目录)' 13/11/29 08:31:00错误exec.Task:作业提交失败'异常'java.io.IOException(Target /tmp/hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib是一个目录)' java.io.IOException:Target/tmp/hadoop-yarn /staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib是位于org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:500) 处的目录 ,位于org.apache.hadoop.fs.FileUtil。 (org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:348) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) at org.apache.hadoop.mapreduce.JobSubmitter.copyRemoteFiles(JobSubmitter.java:139) at org.apache.hadoop.mapreduce.JobSubmitter。 copyAndConfigureFiles(JobSubmitter.java:212) 在org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300) 在org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387) 在org.apache.hadoop.mapreduce。作业$ 10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job $ 10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security .auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java :1265) at org.apache.hadoop.mapred.JobClient $ 1.run(JobClient.java:562) at org.apache.hadoop.mapred.JobClient $ 1.run(Job Client.java:557) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1491) 在org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557) 在org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548) 在有机.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:425) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:144) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) at org.apache。 hadoop.hive.ql.Driver.runInternal(Driver.java:1020) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli。 CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver。 java:413) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(clidriver中。的java:614) 在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法) 在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 在java.lang.reflect.Method.invoke(Method.java:606) 在org.apache.hadoop.util.RunJar.main(RunJar.java:212)

在考虑中的文件夹中确实存在上dfs,至少是“/ tmp/hadoop-yarn/staging”部分,无论我设置它的权限,配置单元或hadoop都会在作业提交时重置它们。真正令人关注的部分是,完整路径似乎是一个生成的文件夹名称,那么为什么软件自己生成的某些东西有问题?为什么这个路径是一个目录的问题?它应该是什么?

编辑: 这是我的工作表,我试图运行查询:查询 : select * from hive_flow_details where node_id = 100 limit 10;

表:

COL_NAME DATA_TYPE评论 ID BIGINT无
flow_versions_id int无
node_id int无
node_name字符串无

请记住,这种情况发生在我尝试使用任何种类的where子句的任何uery上,因为hive会将它转换为MR作业。

+0

你可以发布你试图运行的查询吗? –

回答

0

我最终解决了这个问题。我在我清理过的classpath中发现了冲突的瓶子,从那时起我没有任何问题。

相关问题