Pyspark，编程初始化spark：IllegalArgumentException：缺少应用程序资源

Question

在Python中创建Spark上下文时，出现以下错误。

app_name="my_app" 
master="local[*]" 
sc = SparkContext(appName=app_name, master=master) 

Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource. 
at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241) 
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160) 
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:276) 
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151) 
at org.apache.spark.launcher.Main.main(Main.java:86) 

.... 

pyspark.zip/pyspark/java_gateway.py", line 94, in launch_gateway 
raise Exception("Java gateway process exited before sending the driver its port number") 
Exception: Java gateway process exited before sending the driver its port number 

火花发射器似乎失败了。

Answer 1

这是由于先前存在的env变量发生冲突。我在Python程序中删除了它们，现在它可以顺利运行。

例如：

import os 
#check if pyspark env vars are set and then reset to required or delete. 
del os.environ['PYSPARK_SUBMIT_ARGS']