0
我使用CDH 5.7.2和MrJob提交MapReduce工作MrJob在Hadoop不能导入库
当我尝试在localmode,一切工作正常,但是当我使用-r hadoop
它给了我以下错误:
Task Id : attempt_1471071791922_0005_m_000001_2, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:325)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:538)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
我想通了,发生问题,当我尝试导入库:
import geopy
from geopy.distance import great_circle
这是我如何执行我的脚本:
python test2.py -r hadoop hdfs:///user/dataset/test_data --hadoop-streaming-jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-streaming.jar
P.S. MrJob没有好的hadoop-streaming-jar,所以我手动指定
如何在Hadoop上执行MapReduce jon 库导入?
尝试在执行-libjars增加一个paramater/pathtoyour3rdpartylib – Aditya
我不知道我有任何python库作为jar文件 –