如何读取Pig中的值列表作为包并将其与特定值进行比较？

Question

输入：

IDS：

1111,2222,3333,4444 

员工：

{"name":"abc","id":"1111"} {"name":"xyz","id":"10"} 
{"name":"z","id":"100"} {"name":"m","id":"99"} 
{"name":"pqr","id":"3333"} 

我想筛选员工的ID存在于给定的列表。

预期输出：

{"name":"xyz","id":"10"} {"name":"z","id":"100"} 
{"name":"m","id":"99"} 

现有代码：

idList = LOAD 'pathToFile' USING PigStorage(',') AS (id:chararray); 
empl = LOAD 'pathToFile' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS (data:map[]); 
output = FILTER empl BY data#'id' in (idList); 
-- not working, states: A column needs to be projected from a relation for it to be used as a scalar 
output = FILTER empl BY data#'id' in (idList#id); 
-- not working, states: mismatched input 'id' expecting set null 

Answer 1

JsonLoad()是pig > 0.10本地人，你可以指定模式：

empl = LOAD 'pathToFile' USING JsonLoader('name:chararray, id:chararray'); 
DUMP empl; 

(abc,1111) 
(xyz,10) 
(z,100) 
(m,99) 
(pqr,3333) 

您正在加载idList作为chararray类型的一列表，但您需要一个列表。

加载它作为一个列的表（意味着你修改文件，所以每行只有一条记录）：

idList = LOAD 'pathToFile' USING PigStorage(',') AS (id:chararray); 
DUMP idList; 

(1111) 
(2222) 
(3333) 
(4444) 

或作为一个在线文件，所以没有关系，我们将更改分隔符“T拆分成多列（否则会导致装载只有第一列）：

idList = LOAD 'pathToFile' USING PigStorage(' ') AS (id:chararray); 
idList = FOREACH idList GENERATE FLATTEN(TOKENIZE(id, '[,]')) AS id; 
DUMP idList; 

(1111) 
(2222) 
(3333) 
(4444) 

现在我们可以做一个LEFT JOIN，看看哪些id中不存在的idList然后FILTER只有那些保持。 output是保留字，请不要使用：

res = JOIN empl BY id LEFT, idList BY id; 
res = FILTER res BY idList::id IS NULL; 
DUMP res; 

(xyz,10,) 
(m,99,) 
(z,100,)