Apache Spark：找不到符号错误使用array_contains

Question

我是新来的apache的火花和编写应用程序，通过json文件解析。 json文件中的一个属性是一个字符串数组。如果数组属性不包含字符串“None”，我想运行一个查询来选择一行。我发现了一些使用org.apache.spark.sql.functions包中的array_contains方法的解决方案。然而，当我试图建立我的申请，我得到以下找不到符号错误：

我使用Apache星火2.0，和Maven构建我的项目。那我试图代码进行编译：

import java.util.List; 

import scala.Tuple2; 

import org.apache.spark.SparkConf; 
import org.apache.spark.api.java.JavaPairRDD; 
import org.apache.spark.api.java.JavaRDD; 
import org.apache.spark.api.java.JavaSparkContext; 
import org.apache.spark.sql.Column; 
import org.apache.spark.sql.Dataset; 
import org.apache.spark.sql.Row; 
import org.apache.spark.sql.SQLContext; 
import org.apache.spark.sql.SparkSession; 
import org.apache.spark.sql.functions; 

import static org.apache.spark.sql.functions.col; 
public class temp { 

    public static void main(String[] args) { 
     SparkSession spark = SparkSession 
       .builder() 
       .appName("testSpark") 
       .enableHiveSupport()  
       .getOrCreate(); 

     Dataset<Row> df = spark.read().json("hdfs://XXXXXX.XXX:XXX/project/term/project.json"); 
     df.printSchema(); 
     Dataset<Row> filteredDF = df.select(col("user_id"),col("elite")); 
     df.createOrReplaceTempView("usersTable"); 
     String val[] = {"None"}; 
     Dataset<Row> newDF = df.select(col("user_id"),col("elite").where(array_contains(col("elite"),"None"))); 
     newDF.show(10); 
     JavaRDD<Row> users = filteredDF.toJavaRDD(); 
    } 
} 

下面是我的pom.xml文件：

<project> 
    <modelVersion>4.0.0</modelVersion> 
    <groupId>term</groupId> 
    <artifactId>Data</artifactId> 
    <version>0.0.1</version> 

    <!-- specify java version needed??? --> 
    <properties> 
    <maven.compiler.source>1.8</maven.compiler.source> 
    <maven.compiler.target>1.8</maven.compiler.target> 
    </properties> 

    <!-- overriding src/java directory... --> 
    <build> 
    <sourceDirectory>src/</sourceDirectory> 
    </build> 

    <!-- telling it to create a jar --> 
    <packaging>jar</packaging> 


    <!-- DEPENDENCIES --> 
    <dependencies> 
    <dependency> 
     <groupId>org.apache.hadoop</groupId> 
     <artifactId>hadoop-client</artifactId> 
     <version>2.7.3</version> 
    </dependency> 
    <dependency> 
     <groupId>org.scala-lang</groupId> 
     <artifactId>scala-library</artifactId> 
     <version>2.11.7</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-streaming_2.10</artifactId> 
     <version>2.0.0</version> 
    </dependency> 
    </dependencies> 
</project> 

Answer 1

变化

Dataset<Row> newDF = df.select(col("user_id"),col("elite").where(
     array_contains(col("elite"),"None"))); 

到functions类添加为预选赛，像

Dataset<Row> newDF = df.select(col("user_id"),col("elite").where(
     functions.array_contains(col("elite"),"None"))); 

或，使用Static Import

import static org.apache.spark.sql.functions.array_contains;