前缀除主键列以外的所有火花数据帧列

Question

以下是我用于列名前缀的代码。我想排除一个或多个主键列。我的primaryKeys是一个字符串数组，它可能包含一个或多个主键字段。

val primaryKeys = args(2).split("-") 

val prefix = "w1." 
val renamedColumns = df.columns.map(c=> df(c).as(s"$prefix$c")) 
val dfNew = df.select(renamedColumns: _*) 

val prefix2 = "w2." 
val renamedColumns2 = df2.columns.map(c2=> df2(c2).as(s"$prefix2$c2")) 
val df2New = df2.select(renamedColumns2: _*) 

If it is just one column i was able to rename using withColumnRenamed but i am unable to do it if i have multiple primary columns. 

我不能做这样的事情

for (primaryKey <- primaryKeys) { 
dfNew.withColumnRenamed("$PREFIX1"+s"${primaryKey}",s"$primaryKey").toDF() 
} 

有人能帮忙吗？

Answer 1

如果我正确理解你的问题，你可以有条件地组装renamedColumns到如下前缀只有非主键列：

val df = Seq(
    ("1", "a", "c1", "d1"), 
    ("2", "b", "c2", "d2"), 
    ("3", "c", "c3", "d3") 
).toDF("pk1", "pk2", "col1", "col2") 

val primaryKeys = Array("pk1", "pk2") 
val prefix = "w1." 

val renamedColumns = df.columns.map(
    c => if (primaryKeys contains c) df(c).as(c) else df(c).as(s"$prefix$c") 
) 

val dfNew = df.select(renamedColumns: _*) 

dfNew.show 
+---+---+-------+-------+ 
|pk1|pk2|w1.col1|w1.col2| 
+---+---+-------+-------+ 
| 1| a|  c1|  d1| 
| 2| b|  c2|  d2| 
| 3| c|  c3|  d3| 
+---+---+-------+-------+