Map减少作业生成空输出文件

程序正在生成空输出文件。任何人都可以请建议我我哪里错了。任何帮助将不胜感激。我试图把job.setNumReduceTask（0），因为我不使用reducer但仍输出文件是空的。Map减少作业生成空输出文件

public static class PrizeDisMapper extends Mapper<LongWritable, Text, Text, Pair>{ 
int rating = 0; 
Text CustID; 
IntWritable r; 
Text MovieID; 
public void map(LongWritable key, Text line, Context context 
       ) throws IOException, InterruptedException { 
     String line1 = line.toString(); 
     String [] fields = line1.split(":"); 
     if(fields.length > 1) 
      { 
      String Movieid = fields[0]; 
      String line2 = fields[1]; 
      String [] splitline = line2.split(","); 
      String Custid = splitline[0]; 
      int rate = Integer.parseInt(splitline[1]); 
      r = new IntWritable(rate); 
      CustID = new Text(Custid); 
      MovieID = new Text(Movieid); 
      Pair P = new Pair(); 
      context.write(MovieID,P); 
      } 
      else 
      { 
      return; 
      } 
    } 
} 

public static class IntSumReducer extends Reducer<Text,Pair,Text,Pair> { 
private IntWritable result = new IntWritable(); 
public void reduce(Text key, Iterable<Pair> values, 
        Context context 
        ) throws IOException, InterruptedException { 
    for (Pair val : values) { 
    context.write(key, val); 
    } 
    } 

    public class Pair implements Writable 
    { 
    String key; 
    int value; 
    public void write(DataOutput out) throws IOException { 
    out.writeInt(value); 
    out.writeChars(key); 
    } 
    public void readFields(DataInput in) throws IOException { 
    key = in.readUTF(); 
    value = in.readInt(); 
    } 
    public void setVal(String aKey, int aValue) 
    { 
    key = aKey; 
     value = aValue; 
    }

主要类：

public static void main(String[] args) throws Exception { 
Configuration conf = new Configuration(); 
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); 
if (otherArgs.length != 2) { 
    System.err.println("Usage: wordcount <in> <out>"); 
    System.exit(2); 
} 
Job job = new Job(conf, "word count"); 
job.setJarByClass(WordCount.class); 
job.setMapperClass(TokenizerMapper.class); 
job.setCombinerClass(IntSumReducer.class); 
job.setReducerClass(IntSumReducer.class); 
job.setInputFormatClass (TextInputFormat.class); 
FileInputFormat.addInputPath(job, new Path(otherArgs[0])); 
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); 
job.setOutputKeyClass(Text.class); 
job.setOutputValueClass(Pair.class); 
System.exit(job.waitForCompletion(true) ? 0 : 1);

感谢@Pathmanaban Palsamy和@克里斯Gerken您的建议。我根据您的建议修改了代码，但仍然获取空输出文件。任何人都可以请我建议我在我的主类输入和输出配置。我是否需要在输入中指定Pair类到mapper &如何？

来源

2014-11-04 Manvi

我猜作为

public void reduce(Text key, Iterable<Pair> values, 
       Context context 
       ) throws IOException, InterruptedException

您获得通过一个可迭代（对象从中可以得到一个迭代器），您可以使用它遍历所有的都是值的降低方法应该声明映射到给定的密钥。

来源

2014-11-04 04:47:47

右键dignosis（+1）。看看http://stackoverflow.com/questions/26710866/probelm-with-mapreduce-wordcount-program-output-is-same-as-the-input-file，OP搞砸了2种方法。 – blackSmith 2014-11-04 06:55:47

由于不需要减速，我怀疑下面一行

Pair P = new Pair(); context.write(MovieID,P);

对空将是问题。也请您检查您给正确keyclass和valueclass您的驱动程序类像

job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(Pair.class);

来源

2014-11-04 08:17:59

Map减少作业生成空输出文件

回答

相关问题