MapReduce的矩阵乘法与下面的链接上的hadoop

我试图运行提到的矩阵乘法的例子（源代码）：MapReduce的矩阵乘法与下面的链接上的hadoop

http://www.norstad.org/matrix-multiply/index.html

我在pseudodistributed模式的hadoop设置和我配置它使用此教程：

http://hadoop-tutorial.blogspot.com/2010/11/running-hadoop-in-pseudo-distributed.html?showComment=1321528406255#c3661776111033973764

当我运行我的jar文件，然后我得到以下错误：

Identity test 
11/11/30 10:37:34 INFO input.FileInputFormat: Total input paths to process : 2 
11/11/30 10:37:34 INFO mapred.JobClient: Running job: job_201111291041_0010 
11/11/30 10:37:35 INFO mapred.JobClient: map 0% reduce 0% 
11/11/30 10:37:44 INFO mapred.JobClient: map 100% reduce 0% 
11/11/30 10:37:56 INFO mapred.JobClient: map 100% reduce 100% 
11/11/30 10:37:58 INFO mapred.JobClient: Job complete: job_201111291041_0010 
11/11/30 10:37:58 INFO mapred.JobClient: Counters: 17 
11/11/30 10:37:58 INFO mapred.JobClient: Job Counters 
11/11/30 10:37:58 INFO mapred.JobClient:  Launched reduce tasks=1 
11/11/30 10:37:58 INFO mapred.JobClient:  Launched map tasks=2 
11/11/30 10:37:58 INFO mapred.JobClient:  Data-local map tasks=2 
11/11/30 10:37:58 INFO mapred.JobClient: FileSystemCounters 
11/11/30 10:37:58 INFO mapred.JobClient:  FILE_BYTES_READ=114 
11/11/30 10:37:58 INFO mapred.JobClient:  HDFS_BYTES_READ=248 
11/11/30 10:37:58 INFO mapred.JobClient:  FILE_BYTES_WRITTEN=298 
11/11/30 10:37:58 INFO mapred.JobClient:  HDFS_BYTES_WRITTEN=124 
11/11/30 10:37:58 INFO mapred.JobClient: Map-Reduce Framework 
11/11/30 10:37:58 INFO mapred.JobClient:  Reduce input groups=2 
11/11/30 10:37:58 INFO mapred.JobClient:  Combine output records=0 
11/11/30 10:37:58 INFO mapred.JobClient:  Map input records=4 
11/11/30 10:37:58 INFO mapred.JobClient:  Reduce shuffle bytes=60 
11/11/30 10:37:58 INFO mapred.JobClient:  Reduce output records=2 
11/11/30 10:37:58 INFO mapred.JobClient:  Spilled Records=8 
11/11/30 10:37:58 INFO mapred.JobClient:  Map output bytes=100 
11/11/30 10:37:58 INFO mapred.JobClient:  Combine input records=0 
11/11/30 10:37:58 INFO mapred.JobClient:  Map output records=4 
11/11/30 10:37:58 INFO mapred.JobClient:  Reduce input records=4 
11/11/30 10:37:58 INFO input.FileInputFormat: Total input paths to process : 1 
11/11/30 10:37:59 INFO mapred.JobClient: Running job: job_201111291041_0011 
11/11/30 10:38:00 INFO mapred.JobClient: map 0% reduce 0% 
11/11/30 10:38:09 INFO mapred.JobClient: map 100% reduce 0% 
11/11/30 10:38:21 INFO mapred.JobClient: map 100% reduce 100% 
11/11/30 10:38:23 INFO mapred.JobClient: Job complete: job_201111291041_0011 
11/11/30 10:38:23 INFO mapred.JobClient: Counters: 17 
11/11/30 10:38:23 INFO mapred.JobClient: Job Counters 
11/11/30 10:38:23 INFO mapred.JobClient:  Launched reduce tasks=1 
11/11/30 10:38:23 INFO mapred.JobClient:  Launched map tasks=1 
11/11/30 10:38:23 INFO mapred.JobClient:  Data-local map tasks=1 
11/11/30 10:38:23 INFO mapred.JobClient: FileSystemCounters 
11/11/30 10:38:23 INFO mapred.JobClient:  FILE_BYTES_READ=34 
11/11/30 10:38:23 INFO mapred.JobClient:  HDFS_BYTES_READ=124 
11/11/30 10:38:23 INFO mapred.JobClient:  FILE_BYTES_WRITTEN=100 
11/11/30 10:38:23 INFO mapred.JobClient:  HDFS_BYTES_WRITTEN=124 
11/11/30 10:38:23 INFO mapred.JobClient: Map-Reduce Framework 
11/11/30 10:38:23 INFO mapred.JobClient:  Reduce input groups=2 
11/11/30 10:38:23 INFO mapred.JobClient:  Combine output records=2 
11/11/30 10:38:23 INFO mapred.JobClient:  Map input records=2 
11/11/30 10:38:23 INFO mapred.JobClient:  Reduce shuffle bytes=0 
11/11/30 10:38:23 INFO mapred.JobClient:  Reduce output records=2 
11/11/30 10:38:23 INFO mapred.JobClient:  Spilled Records=4 
11/11/30 10:38:23 INFO mapred.JobClient:  Map output bytes=24 
11/11/30 10:38:23 INFO mapred.JobClient:  Combine input records=2 
11/11/30 10:38:23 INFO mapred.JobClient:  Map output records=2 
11/11/30 10:38:23 INFO mapred.JobClient:  Reduce input records=2 
Exception in thread "main" java.io.IOException: Cannot open filename /tmp/Matrix Multiply/out/_logs 
     at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.ja va:1497) 
     at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.<init>(DFSClient.java :1488) 
     at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:376) 
     at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSyst em.java:178) 
     at org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1 437) 
     at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:142 4) 
     at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:141 7) 
     at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:141 2) 
     at TestMatrixMultiply.fillMatrix(TestMatrixMultiply.java:62) 
     at TestMatrixMultiply.readMatrix(TestMatrixMultiply.java:84) 
     at TestMatrixMultiply.checkAnswer(TestMatrixMultiply.java:108) 
     at TestMatrixMultiply.runOneTest(TestMatrixMultiply.java:144) 
     at TestMatrixMultiply.testIdentity(TestMatrixMultiply.java:156) 
     at TestMatrixMultiply.main(TestMatrixMultiply.java:258) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. java:57) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces sorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:601) 
     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

可有人请建议我，我究竟做错了什么？谢谢

来源

2011-11-30 waqas

它试图读取作业输出。当你提交给你的群集时，它会添加这个_log目录。由于目录没有序列文件，因此无法读取。

您必须更改读取它的代码。

我也照本宣科的东西等于：

FileStatus[] stati = fs.listStatus(output); 
for (FileStatus status : stati) { 
    if (!status.isDir()) { 
     Path path = status.getPath(); 
     // HERE IS THE READ CODE FROM YOUR EXAMPLE 
    } 
}

http://code.google.com/p/hama-shortest-paths/source/browse/trunk/hama-gsoc/src/de/jungblut/clustering/mapreduce/KMeansClusteringJob.java#127

来源

2011-11-30 12:56:46

谢谢托马斯。我说得对。你有什么想法，关于我提到的矩阵乘法的例子，为什么这个工作正常，而且在hadoop独立模式下工作正常，但在检查答案的时候不能与hadoop分布模式一起工作？ – waqas

因为在分布式模式下运行时，_log存储在HDFS中，在伪分布式模式/独立模式下，这将存储在tasktracker日志中。 –

它可能是一个原始的建议，但您可能需要更改日志文件名与的/ tmp /矩阵\乘法/出/ _logs。目录名称中的空格可能不会自动处理，我认为您正在使用Linux。

来源

2011-11-30 10:04:03 miette

Thanks miette。是的，我正在使用Linux，事情是所有这些日志文件和路径都是由程序本身定义的。那么你是否仍然认为它可能与你描述的问题相同？由于我对这个领域相当陌生，所以请纠正我，如果我错了 – waqas

它不是空间，它试图读取一个目录。 –

有在TestMatrixMultiply.java两个问题：

正如托马斯·Jungblut说，_logs应排除在readMatrix（）方法。我已经改变了这样的代码：

if (fs.isFile(path)) { 
     fillMatrix(result, path); 
    } else { 
     FileStatus[] fileStatusArray = fs.listStatus(path); 
     for (FileStatus fileStatus : fileStatusArray) { 
      if (!fileStatus.isDir()) // this line is added by me 
       fillMatrix(result, fileStatus.getPath()); 
     } 
    }

在main（）方法的末尾，fs.delete应该评论，或者输出目录将在MapReduce工作结束后立即每次删除。
```
finally { 
     //fs.delete(new Path(DATA_DIR_PATH), true); 
    } 
```

来源

2012-01-11 07:52:44 saintdevil

MapReduce的矩阵乘法与下面的链接上的hadoop

回答

相关问题