2014-09-20 118 views
0

我想在Windows 7建立星火1.1.0,我跑了sbt assembly与以下java.lang.OutOfMemoryError: Java heap space错误失败:如何在Windows 7上构建Spark 1.1.0?

...  
[warn] Merging 'parquet\schema\Type$Repetition$3.class' with strategy 'first' 
[warn] Merging 'parquet\schema\Type$Repetition.class' with strategy 'first' 
[warn] Merging 'parquet\schema\Type.class' with strategy 'first' 
[warn] Merging 'parquet\schema\TypeConverter.class' with strategy 'first' 
[warn] Merging 'parquet\schema\TypeVisitor.class' with strategy 'first' 
[warn] Merging 'plugin.properties' with strategy 'first' 
[warn] Merging 'plugin.xml' with strategy 'first' 
[warn] Merging 'reference.conf' with strategy 'concat' 
[warn] Merging 'rootdoc.txt' with strategy 'first' 
[warn] Strategy 'concat' was applied to a file 
[warn] Strategy 'discard' was applied to 1723 files 
[warn] Strategy 'first' was applied to 2713 files 
[info] Assembly up to date: D:\app\spark\assembly\target\scala-2.10\spark-assembly-1.1.1-SNAPSHOT-hadoop1.0.4.jar 
[info] Packaging D:\app\spark\examples\target\scala-2.10\spark-examples-1.1.1-SNAPSHOT-hadoop1.0.4.jar ... 
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space 
     at java.util.concurrent.FutureTask.report(Unknown Source) 
     at java.util.concurrent.FutureTask.get(Unknown Source) 
     at sbt.ConcurrentRestrictions$$anon$4.take(ConcurrentRestrictions.scala:196) 
     at sbt.Execute.next$1(Execute.scala:88) 
     ... 
     at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62) 
     at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62) 
     at sbt.Aggregation$$anonfun$evaluatingParser$4$$anonfun$apply$5.apply(Ag 
     ... 
     at xsbt.boot.Boot$.main(Boot.scala:17) 
     at xsbt.boot.Boot.main(Boot.scala) 
Caused by: java.lang.OutOfMemoryError: Java heap space 
     at java.util.Arrays.copyOf(Unknown Source) 
     at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source) 
     at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source) 
     at java.lang.AbstractStringBuilder.append(Unknown Source) 
     at java.lang.StringBuilder.append(Unknown Source) 
     at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:197) 
     at scala.collection.TraversableOnce$$anonfun$addString$1.apply(TraversableOnce.scala:327) 
     at scala.collection.Iterator$class.foreach(Iterator.scala:727) 
     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) 
     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) 
     at scala.collection.AbstractIterable.foreach(Iterable.scala:54) 
     at scala.collection.TraversableOnce$class.addString(TraversableOnce.scala:320) 
     at scala.collection.AbstractTraversable.addString(Traversable.scala:105) 
     at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:286) 
     at scala.collection.AbstractTraversable.mkString(Traversable.scala:105) 
     at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:288) 
     at scala.collection.AbstractTraversable.mkString(Traversable.scala:105) 
     at sbt.Package$.sourcesDebugString(Package.scala:111) 
     at sbt.Package$$anonfun$makeJar$2.apply(Package.scala:106) 
     at sbt.Package$$anonfun$makeJar$2.apply(Package.scala:106) 
     at sbt.MultiLogger.log(MultiLogger.scala:31) 
     at sbt.Logger$class.debug(Logger.scala:121) 
     at sbt.AbstractLogger.debug(Logger.scala:11) 
     at sbt.Package$.makeJar(Package.scala:106) 
     at sbtassembly.Plugin$Assembly$.sbtassembly$Plugin$Assembly$$makeJar$1(Plugin.scala:187) 
     at sbtassembly.Plugin$Assembly$$anonfun$7$$anonfun$apply$6.apply(Plugin.scala:226) 
     at sbtassembly.Plugin$Assembly$$anonfun$7$$anonfun$apply$6.apply(Plugin.scala:222) 
     at sbt.Tracked$$anonfun$outputChanged$1.apply(Tracked.scala:57) 
     at sbt.Tracked$$anonfun$outputChanged$1.apply(Tracked.scala:52) 
     at sbtassembly.Plugin$Assembly$.apply(Plugin.scala:230) 
     at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:373) 
     at sbtassembly.Plugin$Assembly$$anonfun$assemblyTask$1.apply(Plugin.scala:370) 
[error] java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Jav 
a heap space 
[error] Use 'last' for the full log. 

我尝试添加-Xmx1024msbt.bat文件,但它并没有解决问题。

回答

1

从日志中看来,您正试图从源代码构建1.1.1-SNAPSHOT版本。

[info] Assembly up to date: D:\app\spark\assembly\target\scala-2.10\spark-assembly-1.1.1-SNAPSHOT-hadoop1.0.4.jar

如果你想使用星火我会强烈建议从http://spark.apache.org/downloads.html下载包为您的操作系统来代替。

如果你坚持从源头构建Spark,你应该考虑将构建工具从sbt改为Apache Maven。在Spark文档中给出了GitHub上的Building SparkBuilding Spark with Maven,它们似乎已经切换到Apache Maven作为最终用户的构建工具。

使用Maven构建Spark和遵循的方针:

You’ll need to configure Maven to use more memory than usual by setting MAVEN_OPTS. We recommend the following settings:

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" 

还应与SBT工作,虽然。使用SBT_OPTS并重新开始。

相关问题