在PIG中加载多个文件

我有35个Csv文件我想用Pig加载数据。我有以下尝试在PIG中加载多个文件

1) A = LOAD '/home/mrinmoy/Desktop/Sampath Project/Household/{HLPCA-00000,HLPCA-01000,HLPCA-02000,HLPCA-03000,HLPCA-04000,HLPCA-05000,HLPCA-06000,HLPCA-07000,HLPCA-08000,HLPCA-09000,HLPCA-10000,HLPCA-11000,HLPCA-12000,HLPCA-13000,HLPCA-14000,HLPCA-15000,HLPCA-16000,HLPCA-17000,HLPCA-18000,HLPCA-19000,HLPCA-20000,HLPCA-21000,HLPCA-22000,HLPCA-23000,HLPCA-24000,HLPCA-25000,HLPCA-26000,HLPCA-27000,HLPCA-28000,HLPCA-29000,HLPCA-30000,HLPCA-31000,,HLPCA-32000,,HLPCA-33000,,HLPCA-34000,,HLPCA-35000}.csv' UsingPigStorage(',');

对于这个尝试我已经得到了错误

014-10-06 00:32:07,130 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. Can not create a Path from an empty string 
Details at logfile: /home/mrinmoy/Desktop/Sampath Project/Household/pig_1412580582549.log

在接下来的尝试，我已经改变脚本使用SomeLoader（）试了一下;

2) A = LOAD '/home/mrinmoy/Desktop/Sampath Project/Household/{HLPCA-00000,HLPCA-01000,HLPCA-02000,HLPCA-03000,HLPCA-04000,HLPCA-05000,HLPCA-06000,HLPCA-07000,HLPCA-08000,HLPCA-09000,HLPCA-10000,HLPCA-11000,HLPCA-12000,HLPCA-13000,HLPCA-14000,HLPCA-15000,HLPCA-16000,HLPCA-17000,HLPCA-18000,HLPCA-19000,HLPCA-20000,HLPCA-21000,HLPCA-22000,HLPCA-23000,HLPCA-24000,HLPCA-25000,HLPCA-26000,HLPCA-27000,HLPCA-28000,HLPCA-29000,HLPCA-30000,HLPCA-31000,,HLPCA-32000,,HLPCA-33000,,HLPCA-34000,,HLPCA-35000}.csv' using SomeLoader();

但我得到的错误说这个

2014-10-06 00:39:42,905 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve SomeLoader using imports: [, org.apache.pig.builtin., org.apache.pig.impl.builtin.] 
Details at logfile: /home/mrinmoy/Desktop/Sampath Project/Household/pig_1412580912789.log

来源

2014-10-06 G Krishna Sampath

猪总是会加载一个目录中的所有文件。所以你只需要用你的CSV文件指定目录。

A = LOAD '/home/mrinmoy/Desktop/Sampath Project/Household/' using PigStorage(',');

另外请注意usingPigStorage()缺少一个空格。它应该是using PigStorage()。而你有一些双逗号：...HLPCA-31000,,HLPCA-32000,,HLPCA-33000,,HLPCA-34000,,HLPCA-35000}...

来源

2014-10-06 07:44:15 0xMB

猪支持提供文件名作为正则表达式。所以，你可以提供类似：

A = LOAD '/home/mrinmoy/Desktop/Sampath Project/Household/HLPCA*' Using PigStorage(',');

，它会加载与来自家庭目录“HLPCA”开头的所有文件。

来源

2014-10-06 13:27:29 Amar

在PIG中加载多个文件

回答

相关问题