SPARK Mlllib - <console>：37：错误：值筛选器不是长的成员

Question

嗨， 我有一个数据集，通过Transaction_ID组所有产品，我想排除Transaction_ID少于两种产品。对于我使用这个：

val edges = df.groupBy(col("Transaction_ID")).agg(collect_list(col("Product_ID")) as "Product_ID").withColumn("Product_ID", concat_ws(",", col("Product_ID"))).count().filter("count >= 2")

但是，当我执行此我得到这个错误：

<console>:37: error: value filter is not a member of Long 

我怎样才能解决这个问题？

非常感谢！

Answer 1

您可以尝试如下。

val df = Seq(("tx-1", "aaa"), ("tx-2", "bbb"), ("tx-1", "ccc"),("tx-4", "ccc")).toDF("Transaction_ID", "Product_ID") 
df.show 

+--------------+----------+ 
|Transaction_ID|Product_ID| 
+--------------+----------+ 
|   tx-1|  aaa| 
|   tx-2|  bbb| 
|   tx-1|  ccc| 
|   tx-4|  ccc| 
+--------------+----------+ 

如果你想TRANSACTION_ID只有这样，你可以使用

val df4 =df.groupBy(col("Transaction_ID")).count().filter(col("count") >= 2) 
df4.show 

如果你想同时TRANSACTION_ID和PRODUCT_ID然后

val df1 = df.groupBy(col("Transaction_ID")).count().filter(col("count") >= 2) 
val df2 = df.groupBy(col("Transaction_ID")).agg(collect_list(col("Product_ID")) as "Product_ID").withColumn("Product_ID", concat_ws(",", col("Product_ID"))) 
val df3 = df1.join(df2, df1("Transaction_ID") === df2("Transaction_ID"), "inner").select(df2("Transaction_ID"),df2("Product_ID")) 
df3.show 

+--------------+----------+ 
|Transaction_ID|Product_ID| 
+--------------+----------+ 
|   tx-1| aaa,ccc| 
+--------------+----------+