Set the producer's transactional.id to a stable, unique string per logical producer instance; this enables idempotent delivery and transaction coordination.
Call initTransactions() once at startup, then wrap each batch of sends in beginTransaction() / commitTransaction(), using abortTransaction() in the catch block on failure.
On the consumer side, set isolation.level=read_committed so that transactional messages are only visible after the producer commits; aborted transaction records are filtered out.
With EOS v2 (Kafka 2.5+), the transaction coordinator epoch mechanism prevents zombie producers from committing stale transactions; ensure all broker versions support EOS v2 (epoch fencing).
For Kafka Streams or Kafka Connect sink connectors, set processing.guarantee=exactly_once_v2 (Streams) or the sink connector's exactly-once flag to enable the framework-level transaction management.
Monitor for incomplete transactions via consumer lag on the __transaction_state internal topic and alert on aborted transaction rates.
Known gotchas
transactional.id must be stable across producer restarts; using a random ID defeats epoch fencing and loses exactly-once guarantees on restart.
Exactly-once applies to the Kafka layer only; if the downstream sink (database, file system) does not support idempotent writes, end-to-end exactly-once requires additional deduplication logic.
Large transaction batches that exceed transaction.max.timeout.ms will be aborted by the broker; tune the timeout to be longer than your worst-case batch assembly time.
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp