MapReduce on Cassandra

Question

我正在研究一个简单的MapReduce程序来读取Cassandra列族中的数据，但遇到以下错误。任何有关如何进行的提示都非常感谢。提前致谢！

Cassandra version : 1.0.3 
Hadoop version : 0.20.2 
HADOOP_CLASSPATH has: apache-cassandra-1.0.3.jar, libthrift-0.6.jar, commons-lang-2.4.jar and guava-10.0.1.jar 
What works : Hadoop MR word count example, Reads from Cassandra column family using cassandra-cli, Thrift and Hector 

错误：

11/12/01 20:05:23 INFO mapred.JobClient: Running job: job_201112010835_0009<br/> 
11/12/01 20:05:24 INFO mapred.JobClient: map 0% reduce 0%<br/> 
11/12/01 20:05:33 INFO mapred.JobClient: Task Id : attempt_201112010835_0009_m_000000_0, Status : FAILED<br/> 
Error: java.lang.ClassNotFoundException: com.google.common.collect.AbstractIterator<br/> 
     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)<br/> 
     at java.security.AccessController.doPrivileged(Native Method)<br/> 
     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)<br/> 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)<br/> 
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)<br/> 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)<br/> 
     at java.lang.ClassLoader.defineClass1(Native Method)<br/> 
     at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)<br/> 
     at java.lang.ClassLoader.defineClass(ClassLoader.java:615)<br/> 
     at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)<br/> 
     at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)<br/> 
     at java.net.URLClassLoader.access$000(URLClassLoader.java:58)<br/> 
     at java.net.URLClassLoader$1.run(URLClassLoader.java:197)<br/> 
     at java.security.AccessController.doPrivileged(Native Method)<br/> 
     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)<br/> 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)<br/> 
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)<br/> 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)<br/> 
     at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.initialize(ColumnFamilyRecordReader.java:158)<br/> 
     at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)<br/> 
     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620)<br/> 
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)<br/> 
     at org.apache.hadoop.mapred.Child.main(Child.java:170)<br/> 

Answer 1

你添加卡桑德拉库到classpath您的所有任务服务器的？从维基页面http://wiki.apache.org/cassandra/HadoopSupport：

One configuration note on getting the task trackers to be able to perform queries over 
Cassandra: you'll want to update your HADOOP_CLASSPATH in your <hadoop>/conf/hadoop-env.sh 
to include the Cassandra lib libraries. For example you'll want to do something like 
this in the hadoop-env.sh on each of your task trackers: 


export HADOOP_CLASSPATH=/opt/cassandra/lib/*:$HADOOP_CLASSPATH 

在该示例中的路径应明显与正确的路径系统上的卡桑德拉库所取代。

Answer 2

1）您必须确保您用于Hadoop的所有Cassandra jars（您在cassandra二进制安装的lib目录中找到的所有jar）的版本与Hadoop版本兼容。

2）另外卡桑德拉瓶可能有一些依赖关系，你应该有这个依赖瓶的正确版本。

3）我遇到同样的问题，因为从Hadoop的classpath中清除所有被卡珊德拉罐子，然后从卡桑德拉的最新版本加入所有罐子后jars.Resolved它的这种版本不匹配的。