我想使用listOfWords文件来只计算那些来自任何输入文件的单词。作为FileNotFound获取错误,即使我已经验证该文件在HDFS中的适当位置。Hadoop分布式缓存抛出FileNotFound错误
内部驱动器:
Configuration conf = new Configuration();
DistributedCache.addCacheFile(new URI("/user/training/listOfWords"), conf);
Job job = new Job(conf,"CountEachWord Job");
里面映射:
private Path[] ref_file;
ArrayList<String> globalList = new ArrayList<String>();
public void setup(Context context) throws IOException{
this.ref_file = DistributedCache.getLocalCacheFiles(context.getConfiguration());
FileSystem fs = FileSystem.get(context.getConfiguration());
FSDataInputStream in_file = fs.open(ref_file[0]);
System.out.println("File opened");
BufferedReader br = new BufferedReader(new InputStreamReader(in_file));//each line of reference file
System.out.println("BufferReader invoked");
String eachLine = null;
while((eachLine = br.readLine()) != null)
{
System.out.println("eachLine is: "+ eachLine);
globalList.add(eachLine);
}
}
错误消息:
hadoop jar CountOnlyMatchWords.jar CountEachWordDriver Rhymes CountMatchWordsOut1
Warning: $HADOOP_HOME is deprecated.
14/10/07 22:28:59 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/10/07 22:28:59 INFO input.FileInputFormat: Total input paths to process : 1
14/10/07 22:28:59 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/10/07 22:28:59 WARN snappy.LoadSnappy: Snappy native library not loaded
14/10/07 22:29:00 INFO mapred.JobClient: Running job: job_201409300531_0041
14/10/07 22:29:01 INFO mapred.JobClient: map 0% reduce 0%
14/10/07 22:29:14 INFO mapred.JobClient: Task Id : attempt_201409300531_0041_m_000000_0, Status : FAILED
java.io.FileNotFoundException: File does not exist: /home/training/hadoop-temp/mapred/local /taskTracker/distcache/5910352135771601888_2043607380_1633197895/localhost/user/training/listOfWords
我已经验证了上述文件存在,HDFS。我也尝试使用localRunner。仍然没有工作。
相反DistributedCache.addCacheFile的(新的URI(“/用户工作/ training/listOfWords“),conf);试试这个DistributedCache.addCacheFile(new URI(“/ user/training/listOfWords”),job.getConfiguration()); – user3484461 2014-10-09 06:23:50