从Hive中的多个表中选择增量数据

Question

我在Hive数据库中有五个表（A，B，C，D，E），我必须根据列“id”上的逻辑来合并这些表中的数据。

的条件是：

Select * from A 
UNION 
select * from B (except ids not in A) 
UNION 
select * from C (except ids not in A and B) 
UNION 
select * from D(except ids not in A,B and C) 
UNION 
select * from E(except ids not in A,B,C and D) 

有这个数据插入到决赛桌。

一种方法是创建一个目标表（目标）并为其添加每个UNION阶段的数据，然后使用此表与另一个UNION阶段进行连接。

这将是我.hql文件的一部分：

insert into target 
(select * from A 
UNION 
select B.* from 
A 
RIGHT OUTER JOIN B 
on A.id=B.id 
where ISNULL(A.id)); 

INSERT INTO target 
select C.* from 
target 
RIGHT outer JOIN C 
ON target.id=C.id 
where ISNULL(target.id); 

INSERT INTO target 
select D.* from 
target 
RIGHT OUTER JOIN D 
ON target.id=D.id 
where ISNULL(target.id); 

INSERT INTO target 
select E.* from 
target 
RIGHT OUTER JOIN E 
ON target.id=E.id 
where ISNULL(target.id); 

---

是否有更好的做到这一点？我认为无论如何，我们必须做 多个加入/查找。我很期待的最好的方法来实现这一 在

1）蜂巢与TEZ

2）火花-SQL

许多在此先感谢

Answer 1

如果id在每个表格中都是唯一的，则可以使用row_number代替rank。

select  * 

from  (select  * 
         ,rank() over 
         (
          partition by id 
          order by  src 
         )       as rnk 

      from  (   
            select 1 as src,* from a 
         union all select 2 as src,* from b 
         union all select 3 as src,* from c 
         union all select 4 as src,* from d 
         union all select 5 as src,* from e 
         ) t 
      ) t 

where  rnk = 1 
; 

Answer 2

我想我会尝试这样做的：

with ids as (
     select id, min(which) as which 
     from (select id, 1 as which from a union all 
      select id, 2 as which from b union all 
      select id, 3 as which from c union all 
      select id, 4 as which from d union all 
      select id, 5 as which from e 
      ) x 
    ) 
select a.* 
from a join ids on a.id = ids.id and ids.which = 1 
union all 
select b.* 
from b join ids on b.id = ids.id and ids.which = 2 
union all 
select c.* 
from c join ids on c.id = ids.id and ids.which = 3 
union all 
select d.* 
from d join ids on d.id = ids.id and ids.which = 4 
union all 
select e.* 
from e join ids on e.id = ids.id and ids.which = 5;