1
运行pyspark我有Windows 10有继起的说明安装火花:不能在Jupyter
https://hernandezpaul.wordpress.com/2016/01/24/apache-spark-installation-on-windows-10/
现在,我打开我的jupyter记事本,然后键入以下内容:
import os
import sys
# Path for spark source folder
os.environ['SPARK_HOME']="c:\\Spark"
# Append pyspark to Python Path
sys.path.append("C:\\Spark")
sys.path.append("C:\\Spark\\python")
sys.path.append("C:\\Spark\\python\\lib")
sys.path.append("C:\\Spark\\python\\lib\\py4j-0.9-src.zip")
from pyspark import SparkContext
from pyspark import SparkConf
和似乎cannoot加载累加器库,因为我得到以下错误:
--------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-54-68cce399fff2> in <module>()
12 sys.path.append("C:\\Spark\\python\\pyspark")
13
---> 14 from pyspark import SparkContext
15 from pyspark import SparkConf
16
C:\Spark\python\pyspark\__init__.py in <module>()
39
40 from pyspark.conf import SparkConf
---> 41 from pyspark.context import SparkContext
42 from pyspark.rdd import RDD
43 from pyspark.files import SparkFiles
C:\Spark\python\pyspark\context.py in <module>()
26 from tempfile import NamedTemporaryFile
27
---> 28 from pyspark import accumulators
29 from pyspark.accumulators import Accumulator
30 from pyspark.broadcast import Broadcast
ImportError: cannot import name accumulators
这些是我的sys.path的样子,我以为是正确的文件夹:
['',
'C:\\Anaconda2\\python27.zip',
'C:\\Anaconda2\\DLLs',
'C:\\Anaconda2\\lib',
'C:\\Anaconda2\\lib\\plat-win',
'C:\\Anaconda2\\lib\\lib-tk',
'C:\\Anaconda2',
'c:\\anaconda2\\lib\\site-packages\\sphinx-1.3.5-py2.7.egg',
'c:\\anaconda2\\lib\\site-packages\\setuptools-20.3-py2.7.egg',
'C:\\Anaconda2\\lib\\site-packages',
'C:\\Anaconda2\\lib\\site-packages\\win32',
'C:\\Anaconda2\\lib\\site-packages\\win32\\lib',
'C:\\Anaconda2\\lib\\site-packages\\Pythonwin',
'C:\\Anaconda2\\lib\\site-packages\\IPython\\extensions',
'C:\\Users\\Manuel\\.ipython',
'C:\\Spark',
'C:\\Spark\\python',
'C:\\Spark\\python\\lib',
'C:\\Spark\\python\\lib\\py4j-0.9-src.zip',
'C:\\Spark\\python\\pyspark']
任何帮助将非常感激。
谢谢!