Write Flink SQL table definitions with Kafka source and Iceberg sink using the Table API for a streaming ETL job

domain: nightlies.apache.org/flink · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Create a StreamTableEnvironment from a StreamExecutionEnvironment and configure the catalog to point to an Iceberg catalog (REST, Hive, or Hadoop)
  2. Define the Kafka source table with CREATE TABLE using the 'kafka' connector, specifying topic, bootstrap.servers, format (json or avro), and scan.startup.mode
  3. Define the Iceberg sink table with CREATE TABLE using the 'iceberg' connector and properties for catalog-name, catalog-type, database, and table
  4. Write a streaming INSERT INTO iceberg_table SELECT ... FROM kafka_table with optional transformations, filtering, or aggregations in the SQL body
  5. Call tableEnv.executeSql() to submit the streaming job; Flink compiles the SQL plan and runs it as a continuous DataStream job
  6. Configure checkpointing on the underlying StreamExecutionEnvironment before creating the TableEnvironment to ensure Iceberg sink commits are transactional

Known gotchas

Related routes

Write a Flink SQL pipeline using the upsert-kafka connector as a sink to maintain a compacted changelog stream for downstream consumers
flink.apache.org · 5 steps · unrated
Write Flink SQL temporal joins and lookup joins for stream enrichment
nightlies.apache.org/flink · 6 steps · unrated
Configure the Flink SQL upsert-kafka connector for changelog streams
nightlies.apache.org/flink · 6 steps · unrated

Give your agent this knowledge — and 15,600+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans