当前位置：首页 > news >正文

数据湖技术之Iceberg-03 Iceberg整合Flink 实时写入与增量读取

news 2025/7/9 17:36:14

文章目录

数据湖技术
- 数据湖技术之Iceberg
- - Flink 与 Iceberg 整合
  - - 1. DataFrame API 操作 Iceberg
    - - DataStream API实时写入Iceberg表
      - DataStream API 批量/实时读取 Iceberg 表
      - 指定基于快照实时增量读取数据
      - 合并data files
    - 2. SQL API 操作 Iceberg
    - - SQL API 创建 Iceberg 表并写入数据
      - SQL API 批量查询 Iceberg 表数据
      - SQL API 实时查询 Iceberg 表数据
      - SQL API 指定基于快照实时增量读取数据
    - 3. SQL API读取kafka数据实时写入Iceberg表
  - Iceberg 与 Hudi 对比
  - - 1. 共同点
    - 2. 不同点

数据湖技术

数据湖技术之Iceberg

Flink 与 Iceberg 整合

Flink 版本：1.14.4
iceberg：0.13.2
Maven依赖：

<!-- Flink与Iceberg整合的依赖包-->
<dependency><groupId>org.apache.iceberg</groupId><artifactId>iceberg-flink-runtime-1.14</artifactId><version>${iceberg.version}</version>
</dependency>
<!-- 开发Flink程序依赖包-->
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-java</artifactId><version>${flink.version}</version>
</dependency>
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-streaming-java_${scala.version}</artifactId><version>${flink.version}</version>
</dependency>
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-clients_2.12</artifactId><version>${flink.version}</version>
</dependency>
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-streaming-scala_${scala.version}</artifactId><version>${flink.version}</version>
</dependency>
<!-- Flink Kafka连接器依赖包-->
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-connector-kafka_${scala.version}</artifactId><version>${flink.version}</version>
</dependency>
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-connector-base</artifactId><version>${flink.version}</version>
</dependency>
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-csv</artifactId><version>${flink.version}</version>
</dependency>
<!-- 读取 hdfs 文件需要 jar 包-->
<dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>${hadoop.version}</version>
</dependency>
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-table-common</artifactId><version>${flink.version}</version>
</dependency>
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-table-api-java</artifactId><version>${flink.version}</version>
</dependency>
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-table-api-java-bridge_${scala.version}</artifactId><version>${flink.version}</version>
</dependency>
<dependency><groupId>org.apache.flink</groupId><artifactId>flink-table-planner_${scala.version}</artifactId><version>${flink.version}</version>