星火 - 例外在线程“主要” java.lang.NoClassDefFoundError：组织/阿帕奇/火花/ SQL /数据帧

Question

使用此命令在mobaxterm运行我的火花申请后：

/opt/spark-2.0.0-bin-hadoop2.6/bin/spark-submit --class CDR.SQL cdr-maven-1.0-SNAPSHOT.jar 

出现此错误：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/DataFrame 
     at java.lang.Class.getDeclaredMethods0(Native Method) 
     at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) 
     at java.lang.Class.privateGetMethodRecursive(Class.java:3048) 
     at java.lang.Class.getMethod0(Class.java:3018) 
     at java.lang.Class.getMethod(Class.java:1784) 
     at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:713) 
     at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) 
     at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) 
     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) 
     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.DataFrame 
     at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
     at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
     ... 10 more 

内置了与Maven（pom.xml中）：

<?xml version="1.0" encoding="UTF-8"?> 
<project xmlns="http://maven.apache.org/POM/4.0.0" 
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 

    <groupId>cdr-maven</groupId> 
    <artifactId>cdr-maven</artifactId> 
    <version>1.0-SNAPSHOT</version> 

    <repositories> 
     <repository> 
      <id>apache-repo</id> 
      <name>Apache Repository</name> 
      <url>https://repository.apache.org/content/repositories/releases</url> 
      <releases> 
       <enabled>true</enabled> 
      </releases> 
      <snapshots> 
       <enabled>false</enabled> 
      </snapshots> 
     </repository> 
    </repositories> 

    <build> 
     <plugins> 
      <plugin> 
       <groupId>org.scala-tools</groupId> 
       <artifactId>maven-scala-plugin</artifactId> 
       <executions> 
        <execution> 
         <goals> 
          <goal>compile</goal> 
          <goal>testCompile</goal> 
         </goals> 
        </execution> 
       </executions> 
      </plugin> 
      <plugin> 
       <artifactId>maven-compiler-plugin</artifactId> 
       <configuration> 
        <source>1.7</source> 
        <target>1.7</target> 
       </configuration> 
      </plugin> 
     </plugins> 
    </build> 

    <dependencies> 
     <dependency> 
      <groupId>org.scala-lang</groupId> 
      <artifactId>scala-library</artifactId> 
      <version>2.11.8</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.11</artifactId> 
      <version>1.4.1</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-mllib_2.11</artifactId> 
      <version>1.4.1</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-sql_2.11</artifactId> 
      <version>1.4.1</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-hive_2.11</artifactId> 
      <version>1.4.1</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-hive_2.11</artifactId> 
      <version>1.4.1</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-streaming_2.11</artifactId> 
      <version>1.4.1</version> 
      <scope>provided</scope> 
     </dependency> 

    </dependencies> 

</project> 

我有一种感觉，这是因为我的.jar丢失数据框架依赖，但我不知道如何解决它。

感谢您的帮助。

Answer 1

您正在使用Spark 1.4并将其部署到Spark 2.0群集上，其中类DataFrame已被删除。目前只有DataFrame = Dataset[Row]。请将依赖版本更改为2.0或将群集版本更改为< 2.0

Answer 2

转到“p =项目设置” - >“模块”“相关性”选项卡。更改org.apache.spark的范围：spark_core_2.11以进行编译。