Spark2.1.0不兼容的版本，杰克逊2.7.6

我试图运行的IntelliJ一个简单的例子火花，但我得到这样的错误：Spark2.1.0不兼容的版本，杰克逊2.7.6

Exception in thread "main" java.lang.ExceptionInInitializerError 
at org.apache.spark.SparkContext.withScope(SparkContext.scala:701) 
at org.apache.spark.SparkContext.textFile(SparkContext.scala:819) 
at spark.test$.main(test.scala:19) 
at spark.test.main(test.scala) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) 
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.7.6 
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:64) 
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:19) 
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:730) 
at org.apache.spark.rdd.RDDOperationScope$.<init>(RDDOperationScope.scala:82) 
at org.apache.spark.rdd.RDDOperationScope$.<clinit>(RDDOperationScope.scala)

我尝试更新我的杰克逊的依赖，但它似乎不工作，我这样做：

libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" 
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"

但它仍然出现相同的错误信息，可以有人帮我修复错误？

这里是火花示例代码：

object test { 
def main(args: Array[String]): Unit = { 
    if (args.length < 1) { 
     System.err.println("Usage: <file>") 
     System.exit(1) 
    } 

    val conf = new SparkConf() 
    val sc = new SparkContext("local","wordcount",conf) 
    val line = sc.textFile(args(0)) 

    line.flatMap(_.split(" ")).map((_, 1)).reduceByKey(_+_).collect().foreach(println) 

    sc.stop() 
    } 
}

这里是我的built.sbt：

name := "testSpark2" 

version := "1.0" 

scalaVersion := "2.11.8" 


libraryDependencies += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" 
libraryDependencies += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7" 

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.10" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0"

来源

2017-05-08 Yang

星火2.1.0包含com.fasterxml.jackson.core作为传递依赖。所以，我们不需要包含在libraryDependencies中。

但是，如果你想添加不同的com.fasterxml.jackson.core依赖关系的版本，那么你必须重写它们。就像这样：

name := "testSpark2" 

version := "1.0" 

scalaVersion := "2.11.8" 


dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7" 
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7" 
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7" 

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0" 
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0"

因此，改变你的build.sbt像上面的，并预期它会奏效。

我希望它有帮助！

来源

2017-05-08 10:15:16 himanshuIIITian

谢谢！它适用于我，并解决了这个问题！ – Yang

非常感谢。我是SBT和Maven的新手，一直在解决这个问题。你的解决方案是完美的！ –

谢谢，我错过了杰克逊模块scala覆盖_ $ {scala.major.version} – Val

仅供参考。对于我的情况，我在应用中使用spark和kafka-streams，而kafka-streams使用com.fasterxml.jackson.core 2.8.5。下面添加exclude固定问题

（gradle这个）

compile (group: "org.apache.kafka", name: "kafka-streams", version: "0.11.0.0"){ 
    exclude group:"com.fasterxml.jackson.core" 
}

来源

2017-10-23 06:50:48 Leon

Spark2.1.0不兼容的版本，杰克逊2.7.6

回答

相关问题