大数据全系列 教程
1869个小节阅读:465.9k
目录
408考研
JAVA全系列 教程
面向对象的程序设计语言
Python全系列 教程
Python3.x版本,未来主流的版本
人工智能 教程
顺势而为,AI创新未来
大厂算法 教程
算法,程序员自我提升必经之路
C++ 教程
一门通用计算机编程语言
微服务 教程
目前业界流行的框架组合
web前端全系列 教程
通向WEB技术世界的钥匙
大数据全系列 教程
站在云端操控万千数据
AIGC全能工具班
A A
White Night
Fetch抓取是指Hive在对某些查询操作可以不必使用MapReduce计算,只需要读取指定表对应存储目下的文件,然后输出查询结果到控制台即可。
在hive-default.xml.template文件中hive.fetch.task.conversion,默认值是more,
xxxxxxxxxx
<property>
<name>hive.fetch.task.conversion</name>
<value>more</value>
<description>
Expects one of [none, minimal, more].
Some select queries can be converted to single FETCH task minimizing latency.
Currently the query should be single sourced not having any subquery and should not have
any aggregations or distincts (which incurs RS), lateral views and joins.
0. none : disable hive.fetch.task.conversion
1. minimal : SELECT STAR, FILTER on partition columns, LIMIT only
2. more : SELECT, FILTER, LIMIT only (support TABLESAMPLE and virtual columns)
</description>
</property>
set hive.fetch.task.conversion=none/minimal/more(默认值);
默认做了优化,以下两种情况都不经过mr,改为none后,将走mr。
xxxxxxxxxx
hive> set hive.fetch.task.conversion=none;
hive> set hive.fetch.task.conversion;
hive.fetch.task.conversion=none
hive> select * from person;
Query ID = root_20211122131724_35348e32-3611-40f5-8d26-f1c95b531fa7
Total jobs = 1
Launching Job 1 out of 1
......
将该参数的值修改为默认的more:
xxxxxxxxxx
hive> set hive.fetch.task.conversion=more;
以下SQL不会转为Mapreduce来执行
xxxxxxxxxx
hive> select * from person;
OK
person.id person.name person.likes person.address
1 小明1 ["lol","book","movie"] {"beijing":"xisanqi","shanghai":"pudong"}
2 小明2 ["lol","book","movie"] {"beijing":"xisanqi","shanghai":"pudong"}
3 小明3 ["lol","book","movie"] {"beijing":"xisanqi","shanghai":"pudong"}
4 小明4 ["lol","book","movie"] {"beijing":"xisanqi","shanghai":"pudong"}
5 小明5 ["lol","movie"] {"beijing":"xisanqi","shanghai":"pudong"}
6 小明6 ["lol","book","movie"] {"beijing":"xisanqi","shanghai":"pudong"}
7 小明7 ["lol","book"] {"beijing":"xisanqi","shanghai":"pudong"}
8 小明8 ["lol","book"] {"beijing":"xisanqi","shanghai":"pudong"}
9 小明9 ["lol","book","movie"] {"beijing":"xisanqi","shanghai":"pudong"}
Time taken: 1.135 seconds, Fetched: 9 row(s)
hive> select id,name from person where id <5;
OK
id name
1 小明1
2 小明2
3 小明3
4 小明4
Time taken: 1.375 seconds, Fetched: 4 row(s)
hive> select id,name from person where id <5 limit 2;
OK
id name
1 小明1
2 小明2
Time taken: 0.467 seconds, Fetched: 2 row(s)