我在scala中使用Spark 1.6。Spark scala - 嵌套StructType转换为地图
我使用对象在ElasticSearch中创建了一个索引。对象“params”创建为Map [String,Map [String,String]]。例如:
val params : Map[String, Map[String, String]] = ("p1" -> ("p1_detail" -> "table1"), "p2" -> (("p2_detail" -> "table2"), ("p2_filter" -> "filter2")), "p3" -> ("p3_detail" -> "table3"))
这让我看起来像下面的记录:
{
"_index": "x",
"_type": "1",
"_id": "xxxxxxxxxxxx",
"_score": 1,
"_timestamp": 1506537199650,
"_source": {
"a": "toto",
"b": "tata",
"c": "description",
"params": {
"p1": {
"p1_detail": "table1"
},
"p2": {
"p2_detail": "table2",
"p2_filter": "filter2"
},
"p3": {
"p3_detail": "table3"
}
}
}
},
然后我试图以更新值读取Elasticsearch指数。
星火下面的模式读取索引:
|-- a: string (nullable = true)
|-- b: string (nullable = true)
|-- c: string (nullable = true)
|-- params: struct (nullable = true)
| |-- p1: struct (nullable = true)
| | |-- p1_detail: string (nullable = true)
| |-- p2: struct (nullable = true)
| | |-- p2_detail: string (nullable = true)
| | |-- p2_filter: string (nullable = true)
| |-- p3: struct (nullable = true)
| | |-- p3_detail: string (nullable = true)
我的问题是,对象被解读为一个结构。为了管理和轻松更新字段我想要一个Map,因为我对StructType不是很熟悉。
我试图让物体在UDF的地图,但是我有以下错误:
User class threw exception: org.apache.spark.sql.AnalysisException: cannot resolve 'UDF(params)' due to data type mismatch: argument 1 requires map<string,map<string,string>> type, however, 'params' is of struct<p1:struct<p1_detail:string>,p2:struct<p2_detail:string,p2_filter:string>,p3:struct<p3_detail:string>> type.;
UDF代码片段:
val getSubField : Map[String, Map[String, String]] => String = (params : Map[String, Map[String, String]]) => { val return_string = (params ("p1") getOrElse("p1_detail", null.asInstanceOf[String]) return_string }
我的问题:我们怎样才能转换此Struct到一个地图?我已经阅读过看到文档中可用的toMap方法,但无法找到如何使用它(不熟悉隐式参数),因为我是一名scala初学者。
由于提前,
可以请你加UDF代码片段? –
UDF不会帮助很多,因为我只是试图获得一个Map所需的Map [String,Map [String,String]]。 –
val getSubField:Map [String,Map [String,String]] => String =(params:Map [String,Map [String,String]])=> {0128} )getOrElse(“p1_detail”,null.asInstanceOf [字符串]) \t RETURN_STRING }' –