Enable exactly-once source support on an existing Kafka Connect distributed cluster via the two-phase rollout

domain: kafka.apache.org · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. On every worker in the distributed Connect cluster, add exactly.once.source.support=preparing to the worker properties file
  2. Perform a rolling restart of all workers; in 'preparing' mode workers begin fencing old task generations but do not yet enforce exactly-once writes to Kafka
  3. Once all workers report 'preparing' mode (check REST GET /connectors/{name}/status), update every worker config to exactly.once.source.support=enabled
  4. Perform a second rolling restart; workers now use transactions to write source records and their offsets atomically, preventing duplicates on producer failure
  5. Verify: check the internal Connect offsets topic and confirm records include transactional metadata; monitor for transaction abort metrics on the broker
  6. Test failover: kill a worker mid-task and confirm no duplicate records appear in the target topic after the task restarts on another worker

Known gotchas

Related routes

Configure Kafka exactly-once delivery using EOS v2 transactions for producer-to-consumer pipelines
kafka.apache.org · 6 steps · unrated
Implement Kafka Connect exactly-once source support with a transactional producer and offset management inside a custom SourceConnector
kafka.apache.org · 5 steps · unrated
Configure Kafka exactly-once semantics (EOS) for a transactional producer and idempotent pipeline
kafka · 6 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans