无法在Cosmos中使用自定义Mapreduce jar文件

Question

我创建了我自己的Mapreduce jar文件，并使用HDFS shell命令成功在Cosmos'old Hadoop cluster中进行了测试。下一步是在新集群中测试同一个jar，所以我将它上传到我的主文件夹（user/my.username）的 new cluster's HDFS。

当我尝试使用下面的卷曲后启动MapReduce工作，

curl -X POST "http://computing.cosmos.lab.fiware.org:12000/tidoop/v1/user/my.username/jobs" -d '{"jar":"dt.jar","class_name":"DistanceTest","lib_jars":"dt.jar","input":"input","output":"output"}' -H "Content-Type: application/json" -H "X-Auth-Token: xxxxxxxxxxxxxxxxxxx" 

我得到：

{ “成功”： “假”， “错误”：255}

我尝试了不同的jar的路径值，我得到了相同的结果。我是否必须将我的罐子上传到其他地方，或者我是否错过了一些必要的步骤？

Answer 1

代码中存在一个错误，已经在FIWARE Lab的全局实例中修复。

我测试过这一点：

$ curl -X POST "http://computing.cosmos.lab.fiware.org:12000/tidoop/v1/user/myuser/jobs" -d '{"jar":"mrjars/hadoop-mapreduce-examples.jar","class_name":"wordcount","lib_jars":"mrjars/hadoop-mapreduce-examples.jar","input":"testdir","output":"outputdir"}' -H "Content-Type: application/json" -H "X-Auth-Token: mytoken" 
{"success":"true","job_id": "job_1460639183882_0016"} 

请在这种情况下观察，mrjars/hadoop-mapreduce-examples.jar是hdfs:///user/myuser下的相对路径。在此操作中始终使用相对路径。

$ curl -X GET "http://storage.cosmos.lab.fiware.org:14000/webhdfs/v1/user/myuser/mrjars?op=liststatus&user.name=myuser" -H "X-Auth-Token: mytoken" 
{"FileStatuses":{"FileStatus":[{"pathSuffix":"hadoop-mapreduce-examples.jar","type":"FILE","length":270283,"owner":"myuser","group":"myuser","permission":"644","accessTime":1464702215233,"modificationTime":1464702215479,"blockSize":134217728,"replication":3}]}} 

结果：

$ curl -X GET "http://storage.cosmos.lab.fiware.org:14000/webhdfs/v1/user/myuser/outputdir?op=liststatus&user.name=myuser" -H "X-Auth-Token: mytoken" 
{"FileStatuses":{"FileStatus":[{"pathSuffix":"_SUCCESS","type":"FILE","length":0,"owner":"myuser","group":"myuser","permission":"644","accessTime":1464706333691,"modificationTime":1464706333706,"blockSize":134217728,"replication":3},{"pathSuffix":"part-r-00000","type":"FILE","length":128,"owner":"myuser","group":"myuser","permission":"644","accessTime":1464706333264,"modificationTime":1464706333460,"blockSize":134217728,"replication":3}]}} 

我会上传修复到宇宙中回购尽快GitHub的。