在Spark-Submit Script中导入熊猫时出错

Question

我正在使用Spark-Submit，并且在导入包“熊猫”时出现错误。

从笔记本，我可以导入熊猫没有任何问题。

但是，当我提出通过以下文件火花提交：

from pymongo import MongoClient 
import pandas as pd 

mongo_url = "mongodb://..." 

LOGDB = MongoClient(mongo_url).schedule.log_test 
LOGDB.insert({'m': 'Worked Spark Submit Test'}) 

我收到以下错误：

stderr_1461683917691631694文件：

Spark Command: /usr/local/src/spark160master/ibm-java-x86_64-80//bin/java -cp /usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-launcher_2.10-1.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-network-shuffle_2.10-1.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/gson-2.2.4.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/guava-14.0.1.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/Java-WebSocket-1.3.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/ego/spark-ego_2.10-1.6.0.jar:/usr/local/src/spark160master/spark/profile/batch/:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/spark-assembly-1.6.0-hadoop2.6.0.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-api-jdo-3.2.6.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-core-3.2.10.jar:/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/lib/datanucleus-rdbms-3.2.9.jar:/usr/local/src/data-connectors-1.4.1/* -Dspark.eventLog.enabled=true -Dspark.master=spark://yp-spark-dal09-env5-0005:7083 -Dspark.executor.memory=1024m -Dspark.executor.extraLibraryPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.driver.extraClassPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.driver.extraLibraryPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.app.name=test_submit.py -Dspark.executor.extraClassPath=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/libs/*: -Dspark.files=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py -Dspark.eventLog.dir=/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/events -Dspark.files.useFetchCache=false -Dspark.shuffle.service.port=7340 -Xms512m -Xmx512m org.apache.spark.deploy.ego.EGOClusterDriverWrapper {{WORKER_URL}} /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py org.apache.spark.deploy.PythonRunner --primary-py-file test_submit.py 
======================================== 
log4j:ERROR Could not find value for key log4j.appender.FILE 
log4j:ERROR Could not instantiate appender named "FILE". 
16/04/26 10:18:43 INFO deploy.ego.EGOClusterDriverWrapper: Registered signal handlers for [TERM, HUP, INT] 
16/04/26 10:18:44 WARN hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/04/26 10:18:44 INFO apache.spark.SecurityManager: Changing view acls to: s270-18029e40ba8077-05540e1c503a 
16/04/26 10:18:44 INFO apache.spark.SecurityManager: Changing modify acls to: s270-18029e40ba8077-05540e1c503a 
16/04/26 10:18:44 INFO apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(s270-18029e40ba8077-05540e1c503a); users with modify permissions: Set(s270-18029e40ba8077-05540e1c503a) 
16/04/26 10:18:45 INFO spark.util.Utils: Successfully started service 'EGOClusterDriverWrapper-driver-20160426101839-0010-41cca182-f2d6-408c-81c0-ec288530f82e' on port 55398. 
16/04/26 10:18:45 INFO apache.spark.SecurityManager: Changing view acls to: s270-18029e40ba8077-05540e1c503a 
16/04/26 10:18:45 INFO apache.spark.SecurityManager: Changing modify acls to: s270-18029e40ba8077-05540e1c503a 
16/04/26 10:18:45 INFO apache.spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(s270-18029e40ba8077-05540e1c503a); users with modify permissions: Set(s270-18029e40ba8077-05540e1c503a) 
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Fetching file from /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py to /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py 
16/04/26 10:18:45 INFO spark.util.Utils: Copying /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/b862dff8fef77b647561bb50b1ddf66b0c14511a/test_submit.py to /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py 
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Starting the user JAR in a separate Thread 
16/04/26 10:18:45 INFO deploy.ego.EGOClusterDriverWrapper: Waiting for spark context initialization ... 0 
16/04/26 10:18:46 INFO deploy.ego.EGOClusterDriverWrapper: Final app status: 1, exitCode: 15, (reason: User class threw exception: User application exited with 1) 
16/04/26 10:18:46 INFO deploy.ego.EGOClusterDriverWrapper: Sending driver program state to master 
Exception in thread "Driver" org.apache.spark.SparkUserAppException: User application exited with 1 
     at org.apache.spark.deploy.PythonRunner$.main(PythonRunner.scala:88) 
     at org.apache.spark.deploy.PythonRunner.main(PythonRunner.scala) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) 
     at java.lang.reflect.Method.invoke(Method.java:507) 
     at org.apache.spark.deploy.ego.EGOClusterDriverWrapper$$anon$3.run(EGOClusterDriverWrapper.scala:434) 

stdout_1461683917691631694

no extra config 
load default config from : /usr/local/src/spark160master/spark/profile/batch/ 
/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsUTF8String 
Traceback (most recent call last): 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/data/workdir/spark-driver-37662ccf-26eb-458a-b0d6-a7ca52b65ab4/test_submit.py", line 19, in <module> 
    import pandas as pd 
    File "/usr/lib64/python2.7/site-packages/pandas-0.14.1-py2.7-linux-x86_64.egg/pandas/__init__.py", line 6, in <module> 
    from . import hashtable, tslib, lib 
    File "numpy.pxd", line 157, in init pandas.hashtable (pandas/hashtable.c:22150) 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/__init__.py", line 180, in <module> 
    from . import add_newdocs 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module> 
    from numpy.lib import add_newdoc 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 8, in <module> 
    from .type_check import * 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/lib/type_check.py", line 11, in <module> 
    import numpy.core.numeric as _nx 
    File "/gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/__init__.py", line 14, in <module> 
    from . import multiarray 
ImportError: /gpfs/fs01/user/s270-18029e40ba8077-05540e1c503a/.local/lib/python2.7/site-packages/numpy/core/multiarray.so: undefined symbol: PyUnicodeUCS2_AsUTF8String 

我在做什么错？

Answer 1

我无法完全重现您的错误，因为我没有完整的代码spark-submit作业代码。

你正在初始化你的火花上下文吗？

这里是我的代码工作： -

from pymongo import MongoClient

import pandas as pd

from pyspark import SparkContext

if __name__ == "__main__":

sc = SparkContext(appName="MongoDB CON TEST") 
    print("Hello Spark Demo for Mongodb connection") 
    print("Begining MongoCon") 

    mongo_url = "mongodb://ds045252.mlab.com:45252" 

    connection = MongoClient(mongo_url) 

    db = connection["samplemongodb"] 
    # MongoLab has user authentication 
    db.authenticate("username", "password") 
    db.Photos.insert({'m': 'Worked Spark Submit Test Real test'}) 
    print("Insert worked") 
    sc.stop() 

请点击这里查看完整的代码github上有压痕： - https://github.com/charles2588/sparkjobs/blob/master/mongodbtest.py

与此提交作业： -

./spark-submit.sh --vcap credentials/vcap.json --deploy-mode cluster --master https://169.54.219.20 mongodbtest.py

感谢，

查尔斯。

Answer 2

笔记本预先安装了Pandas和NumPy。你的堆栈跟踪显示spark-submit试图从.local加载NumPy，所以这是你自己安装的版本。在spark-submit环境中是否有预装的NumPy可供使用？

该错误可能是由LD_LIBRARY_PATH的错误设置引起的。你是否从笔记本安装了本地的NumPy，现在正试图从spark-submit中使用它？这些是不同的执行环境，它们可能不共享LD_LIBRARY_PATH的相同设置。