2014-10-29 69 views
0

我试图读取使用PIG行动Oozie的工作流中的特定文件模式叫猪时:错误从Oozie的

Oozie的工作流程:

<workflow-app> 

    <fork> 
     <path to ="subWorkflow1" /> 
     <path to ="subWorkflow2" /> 
    </fork> 

<join> 
</workflow-app> 

**subWorkflow1.xml :** 
<subworkflow> 
<action> 
<pig> 
    Calling the pig script load_data_into_tbl.pig 
<params>{Namenode}</params> 
<params>{input Path}</params> 
</pig> 
</action> 
</subworkflow> 

猪脚本:

load '${namenode}/data/filename*.log -- This file is in HDFS. 
.. . ... . 
Store data into <Table_nm> using HCatStorer` 

InputSource : /data/src_folder/20141029/filename*.log 

第一次尝试:

当我尝试从HDFS中的文件夹中读取数据时,Iam能够首次看到PIG执行成功。其余的执行失败。

第二次尝试:

我发现,当我试图在文件夹中的同一个源文件重新运行Oozie的(20141029)我的执行越来越失败。

第三次尝试:

于是,我试图通过重命名我的源文件的文件夹(20141029)中重新运行该工作流。它工作正常。

可能是什么原因?提前致谢。

错误日志:

Pig Stack Trace 
--------------- 
ERROR 2997: Encountered IOException.   org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1413868377323_35233' doesn't exist in RM. 
at  org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:288) 
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145) 
at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321) 
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) 
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) 
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) 
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) 
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) 

java.io.IOException: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1413868377323_35233' doesn't exist in RM. 
at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:288) 
at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:145) 
at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:321) 
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) 
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) 
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) 
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) 
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) 

at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:348) 
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419) 
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532) 
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183) 
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580) 
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:578) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) 
at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:578) 
at org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:633) 
at org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:627) 
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:150) 
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:429) 
at org.apache.pig.PigServer.launchPlan(PigServer.java:1324) 
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1309) 
at org.apache.pig.PigServer.execute(PigServer.java:1299) 
at org.apache.pig.PigServer.executeBatch(PigServer.java:377) 
at org.apache.pig.PigServer.executeBatch(PigServer.java:355) 
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140) 
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202) 
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173) 
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84) 
at org.apache.pig.Main.run(Main.java:478) 
at org.apache.pig.PigRunner.run(PigRunner.java:49) 
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:286) 
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:226) 
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38) 
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:606) 
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:225) 
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) 
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) 
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:415) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) 
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) 
Failing    
Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2] 

回答

0

我解决了这个Issue.Its并不是一个真正的问题。它是Pig的本质,并且有一些Tickets和正在进行的这个Issue.Once数据存在于一个分区中,您不能使用Pig覆盖数据。那就是问题所在。那就是为什么我能够在第一次尝试中成功加载,而不是在这之后。谢谢 !

有用的链接: https://cwiki.apache.org/confluence/display/Hive/HCatalog+UsingHCat

相关问题