4.5.3 distribute by

distribute by：对应MR作业的partition（自定义分区），通常结合sort by一起使用。在某些情况下需要控制特定的行应该到哪个reduce任务中，为了后续的聚合操作。分区有对应reduce任务，有几个分区就有几个reduce任务；否则就看不到distribute by的效果。

实战演示：

先按照部门编号分区，再按照雇员编号的降序排序

xxxxxxxxxx
hive> set mapreduce.job.reduces=4;
hive> insert overwrite local directory
'/opt/distributebyresult'
select * from emp distribute by deptno sort by empno desc;
Query ID = root_20211118111723_4afc790e-a7e8-4d5c-ba7a-8d790a379ea5
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Defaulting to jobconf value of: 4
......
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 4

在node4中查看结果文件：

xxxxxxxxxx
[root@node4 distributebyresult]# pwd
/opt/distributebyresult
[root@node4 distributebyresult]# ll
总用量 8
-rw-r--r-- 1 root root 229 11月 18 11:18 000000_0
-rw-r--r-- 1 root root   0 11月 18 11:18 000001_0
-rw-r--r-- 1 root root 432 11月 18 11:18 000002_0
-rw-r--r-- 1 root root   0 11月 18 11:18 000003_0
[root@node4 distributebyresult]# cat -A 000000_0 
7902^Aford^Aanalyst^A7566^A1981-12-3^A3000.0^A\N^A20$
7876^Aadams^Aclerk^A7788^A1987-5-23^A1100.0^A\N^A20$
7788^Ascott^Aanalyst^A7566^A1987-4-19^A3000.0^A\N^A20$
7566^Ajones^Amanager^A7839^A1981-4-2^A2975.0^A\N^A20$
7369^Asmith^Aclerk^A7902^A1980-12-17^A800.0^A\N^A20$
[root@node4 distributebyresult]# cat -A 000001_0 
[root@node4 distributebyresult]# cat -A 000002_0 
7934^Amiller^Aclerk^A7782^A1982-1-23^A1300.0^A\N^A10$
7900^Ajames^Aclerk^A7698^A1981-12-3^A950.0^A\N^A30$
7844^Aturner^Asalesman^A7698^A1981-9-8^A1500.0^A0.0^A30$
7839^Aking^Apresident^A\N^A1981-11-17^A5000.0^A\N^A10$
7782^Aclark^Amanager^A7839^A1981-6-9^A2450.0^A\N^A10$
7698^Ablake^Amanager^A7839^A1981-5-1^A2850.0^A\N^A30$
7654^Amartin^Asalesman^A7698^A1981-9-28^A1250.0^A1400.0^A30$
7521^Award^Asalesman^A7698^A1981-2-22^A1250.0^A500.0^A30$
7499^Aallen^Asalesman^A7698^A1981-2-20^A1600.0^A300.0^A30$
[root@node4 distributebyresult]# cat -A 000003_0

distribute by分区规则是根据分区字段的hash值与分区数（reduce任务的总数）进行除模后，余数相同的分到一个分区中。

要求：distribute by语句写在sort by语句的前面。