Hadoop Map Reduce读取文本文件

Question

我想写一个MapReduce程序，它可以读取输入文件并将输出写入另一个文本文件。我打算为此使用BufferedReader类。但我真的不知道如何在MapReduce程序中使用它。

有人可以给我一个它的代码片段吗？

P.S.我完全不熟悉Hadoop和MapReduce编程。所以请耐心等待。

预先感谢您。

Answer 1

下面的代码可以帮助你阅读从HDFS文件，并显示在控制台

import java.io.BufferedReader; 
import java.io.InputStreamReader; 

import org.apache.hadoop.conf.Configuration; 
import org.apache.hadoop.fs.FileSystem; 
import org.apache.hadoop.fs.Path; 

public class Cat{ 
    public static void main (String [] args) throws Exception{ 
     try{ 
      Path pt=new Path("hdfs:/path/to/file");//Location of file in HDFS 
      FileSystem fs = FileSystem.get(new Configuration()); 
      BufferedReader br=new BufferedReader(new InputStreamReader(fs.open(pt))); 
      String line; 
      line=br.readLine(); 
      while (line != null){ 
       System.out.println(line); 
       line=br.readLine(); 
      } 
     }catch(Exception e){ 
     } 
    } 
} 

编辑

驱动

public class ReadFile { 

    public static void main(String[] args) throws Exception { 
     Configuration conf = new Configuration(); 
     Job job = new Job(conf, "Read a File"); 


     FileSystem fs = FileSystem.get(conf); 
     job.setOutputKeyClass(Text.class); 
     job.setOutputValueClass(IntWritable.class); 
     if (fs.exists(new Path(args[1]))) 
      fs.delete(new Path(args[1]), true); 
     job.setMapperClass(Map.class); 
     job.setReducerClass(Reduce.class); 

     job.setInputFormatClass(TextInputFormat.class); 
     job.setOutputFormatClass(TextOutputFormat.class); 

     FileInputFormat.addInputPath(job, new Path(args[0])); 
     FileOutputFormat.setOutputPath(job, new Path(args[1])); 
     job.setJarByClass(ReadFile.class);  
     job.waitForCompletion(true); 
    } 

} 

映射

public class Map extends Mapper<LongWritable, Text, Text, IntWritable> { 

    public void setup(Context context) throws IOException{ 
     Path pt=new Path("hdfs:/path/to/file");//Location of file in HDFS 
     FileSystem fs = FileSystem.get(new Configuration()); 
     BufferedReader br=new BufferedReader(new InputStreamReader(fs.open(pt))); 
     String line; 
     line=br.readLine(); 
     while (line != null){ 
      System.out.println(line); 
      line=br.readLine(); 
     } 
    } 
    public void map(LongWritable key, Text value, Context context) 
      throws IOException, InterruptedException { 
     //as your wish 
     } 
    } 
} 

内容

上面的代码可以帮助您从HDFS中读取文本文件。