Configure Flink SQL jobs to use the filesystem connector with partition commit and success file triggers for exactly-once file sink semantics

domain: flink.apache.org · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Create a Flink SQL sink table with CONNECTOR='filesystem', PATH, FORMAT, and PARTITION BY clause defining the partition columns
  2. Set sink.partition-commit.trigger='partition-time' or 'process-time' and sink.partition-commit.delay to control when a partition is considered complete
  3. Set sink.partition-commit.policy.kind='success-file' to write a _SUCCESS marker after commit, or 'metastore' to notify Hive Metastore
  4. For Hive integration, set sink.partition-commit.policy.kind='metastore,success-file' and provide hive-conf-dir so the catalog is updated atomically
  5. Use WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND in the source DDL so partition-time commit uses event time rather than processing time

Known gotchas

Related routes

Configure a Flink JDBC sink for exactly-once delivery to a relational database using XA transactions
nightlies.apache.org/flink · 5 steps · unrated
Write a Flink SQL pipeline using the upsert-kafka connector as a sink to maintain a compacted changelog stream for downstream consumers
flink.apache.org · 5 steps · unrated
Configure the Flink SQL upsert-kafka connector for changelog streams
nightlies.apache.org/flink · 6 steps · unrated

Give your agent this knowledge — and 15,600+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans