Implement Kafka Connect exactly-once source support with a transactional producer and offset management inside a custom SourceConnector
domain: kafka.apache.org · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Set exactly.once.source.support=enabled in the Connect worker properties and roll the distributed cluster through a two-phase enablement if upgrading an existing cluster
Implement a SourceTask that returns source offsets in each SourceRecord so Connect can store them transactionally alongside Kafka produce in the same transaction
Ensure the Connect worker's producer has transaction support by setting transaction.boundary=poll in connector config, which wraps each poll() batch in a single transaction
Set producer.override.transactional.id.prefix in connector config to namespace transactional IDs per connector instance to avoid collisions across connectors
Verify exactly-once guarantees end-to-end by confirming the consumer reading the sink topic has isolation.level=read_committed
Known gotchas
Exactly-once source support requires all Connect workers in the cluster to be upgraded and the feature enabled before any individual connector can use it; mixed clusters silently fall back to at-least-once
If a source task is fenced by a newer generation (e.g., after rebalance), the previous producer's open transaction is aborted; the task must handle ProducerFencedException and restart cleanly without duplicating records
Connectors that buffer records across multiple poll() calls and emit them out of order break the offset monotonicity assumption required for exactly-once delivery
Give your agent this knowledge — and 15,600+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?