目录大数据全系列 教程
1869个小节阅读：467.6k
408考研
JAVA全系列 教程
面向对象的程序设计语言
Python全系列 教程
Python3.x版本，未来主流的版本
人工智能 教程
顺势而为，AI创新未来
大厂算法 教程
算法，程序员自我提升必经之路
C++ 教程
一门通用计算机编程语言
微服务 教程
目前业界流行的框架组合
web前端全系列 教程
通向WEB技术世界的钥匙
大数据全系列 教程
站在云端操控万千数据
AIGC全能工具班
A A
White Night
xxxxxxxxxx<properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> <java.version>1.8</java.version> <spark.version>3.0.0</spark.version> <spark-streaming-kafka.version>3.1.0</spark-streaming-kafka.version> <hbase.version>2.1.0</hbase.version> <redis.version>4.2.3</redis.version> <hutool.version>5.8.2</hutool.version> <h3.version>3.7.2</h3.version> <fastjson.version>1.2.79</fastjson.version> <lombok.version>1.18.24</lombok.version> </properties>
<dependencyManagement> <dependencies> <!--spark 离线任务核心依赖--> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> <version>${spark.version}</version> </dependency> <!-- spark 离线任务核心依赖 --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.12</artifactId> <version>${spark.version}</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_2.12</artifactId> <version>${spark.version}</version> </dependency> <!--spark 实时任务核心依赖--> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.12</artifactId> <version>${spark.version}</version> </dependency> <!--spark streaming 和kafka集成的依赖配置--> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kafka-0-10_2.12</artifactId> <version>${spark-streaming-kafka.version}</version> </dependency> <!-- HBase 客户端依赖 --> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-client</artifactId> <version>${hbase.version}</version> </dependency> <!-- HBase 客户端依赖 --> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-server</artifactId> <version>${hbase.version}</version> </dependency> <dependency> <groupId>org.apache.hbase</groupId> <artifactId>hbase-mapreduce</artifactId> <version>${hbase.version}</version> </dependency> <!-- Redis 客户端依赖 --> <dependency> <groupId>redis.clients</groupId> <artifactId>jedis</artifactId> <version>${redis.version}</version> </dependency> <!-- 工具包 --> <dependency> <groupId>cn.hutool</groupId> <artifactId>hutool-all</artifactId> <version>${hutool.version}</version> </dependency> <!-- 经纬度计算 --> <dependency> <groupId>com.uber</groupId> <artifactId>h3</artifactId> <version>${h3.version}</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>${fastjson.version}</version> </dependency> <!-- 自动插入编辑器和构建工具 --> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>${lombok.version}</version> </dependency> </dependencies> </dependencyManagement>Spark针对Kafka的不同版本，提供了两套整合方案：spark-streaming-kafka-0-8 和 spark-streaming-kafka-0-10，其主要区别如下：
| spark-streaming-kafka-0-8 | spark-streaming-kafka-0-10 | |
|---|---|---|
| Kafka 版本 | 0.8.2.1 or higher | 0.10.0 or higher |
| AP 状态 | Deprecated 从 Spark 2.3.0 版本开始，Kafka 0.8 支持已被弃用 | Stable(稳定版) |
| 语言支持 | Scala, Java, Python | Scala, Java |
| Receiver DStream | Yes | No |
| Direct DStream | Yes | Yes |
| SSL / TLS Support | No | Yes |
| Offset Commit API(偏移量提交) | No | Yes |
| Dynamic Topic Subscription (动态主题订阅) | No | Yes |
xxxxxxxxxx <!--spark 离线任务核心依赖--> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> </dependency> <!-- spark 离线任务核心依赖 --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.12</artifactId> </dependency> <!--spark 实时任务核心依赖--> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming_2.12</artifactId> </dependency> <!--spark streaming 和kafka集成的依赖配置--> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-streaming-kafka-0-10_2.12</artifactId> </dependency>