如何解决AnalysisException：在星火解决属性（S）

Question

val rdd = sc.parallelize(Seq(("vskp", Array(2.0, 1.0, 2.1, 5.4)),("hyd",Array(1.5, 0.5, 0.9, 3.7)),("hyd", Array(1.5, 0.5, 0.9, 3.2)),("tvm", Array(8.0, 2.9, 9.1, 2.5)))) 
val df1= rdd.toDF("id", "vals") 
val rdd1 = sc.parallelize(Seq(("vskp","ap"),("hyd","tel"),("bglr","kkt"))) 
val df2 = rdd1.toDF("id", "state") 
val df3 = df1.join(df2,df1("id")===df2("id"),"left") 

加入操作工作正常 但是当我再次使用DF2我面临的悬而未决的属性错误

val rdd2 = sc.parallelize(Seq(("vskp", "Y"),("hyd", "N"),("hyd", "N"),("tvm", "Y"))) 
val df4 = rdd2.toDF("id","existance") 
val df5 = df4.join(df2,df4("id")===df2("id"),"left") 

ERROR: org.apache.spark.sql.AnalysisException: resolved attribute(s)id#426

Answer 1

正如上文我的评论是关于https://issues.apache.org/jira/browse/SPARK-10925，更具体地说是https://issues.apache.org/jira/browse/SPARK-14948。重新使用该参考文件会在命名时产生歧义，因此您必须克隆df - 例如，请参阅https://issues.apache.org/jira/browse/SPARK-14948中的最后一条评论。