为什么使用案例类编码JSON时出现错误“无法找到存储在数据集中的类型的编码器”？

Question

我已经写了火花的工作：

object SimpleApp { 
    def main(args: Array[String]) { 
    val conf = new SparkConf().setAppName("Simple Application").setMaster("local") 
    val sc = new SparkContext(conf) 
    val ctx = new org.apache.spark.sql.SQLContext(sc) 
    import ctx.implicits._ 

    case class Person(age: Long, city: String, id: String, lname: String, name: String, sex: String) 
    case class Person2(name: String, age: Long, city: String) 

    val persons = ctx.read.json("/tmp/persons.json").as[Person] 
    persons.printSchema() 
    } 
} 

在IDE中，当我运行的主要功能，出现2错误：

Error:(15, 67) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing sqlContext.implicits._ Support for serializing other types will be added in future releases. 
    val persons = ctx.read.json("/tmp/persons.json").as[Person] 
                   ^

Error:(15, 67) not enough arguments for method as: (implicit evidence$1: org.apache.spark.sql.Encoder[Person])org.apache.spark.sql.Dataset[Person]. 
Unspecified value parameter evidence$1. 
    val persons = ctx.read.json("/tmp/persons.json").as[Person] 
                   ^

但在星火壳牌我可以没有任何错误运行此作业。问题是什么？

Answer 1

错误消息说Encoder不能采取Person案例类。

Error:(15, 67) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing sqlContext.implicits._ Support for serializing other types will be added in future releases. 

将案例类的声明移至SimpleApp范围之外。

Answer 2

如果您在SimpleApp中添加sqlContext.implicits._和spark.implicits._（顺序无关紧要），则会出现同样的错误。

删除一个或另一个将是解决方案：

val spark = SparkSession 
    .builder() 
    .getOrCreate() 

val sqlContext = spark.sqlContext 
import sqlContext.implicits._ //sqlContext OR spark implicits 
//import spark.implicits._ //sqlContext OR spark implicits 

case class Person(age: Long, city: String) 
val persons = ctx.read.json("/tmp/persons.json").as[Person] 

与星火2.1.0

测试有趣的是，如果你添加相同的对象implicits两次，你不会有问题。