Configure the Flink Kafka source with isolation.level=read_committed so the consumer only reads messages from committed Kafka transactions, filtering out in-flight or aborted messages
Enable Flink checkpointing with CheckpointingMode.EXACTLY_ONCE and a durable checkpoint storage backend (S3 or similar) to ensure offsets are committed atomically with operator state
Use the KafkaSink connector (not the legacy FlinkKafkaProducer) and set DeliveryGuarantee.EXACTLY_ONCE on the sink builder; this activates the two-phase commit protocol
Configure transaction.timeout.ms on the Kafka sink to be longer than the maximum expected checkpoint interval to prevent Kafka from aborting open transactions before Flink commits
Set the Kafka broker's transaction.max.timeout.ms to match or exceed the Flink transaction timeout; mismatched values cause the broker to reject transactions with a timeout error
Monitor Flink checkpoint metrics (checkpoint duration, checkpoint size, number of failed checkpoints) and Kafka producer metrics (transaction abort rate) to verify end-to-end health
Known gotchas
Exactly-once Kafka sinks hold open Kafka transactions between checkpoints; if a checkpoint takes longer than transaction.timeout.ms, the broker aborts the transaction and the job fails — always set timeouts conservatively
Scaling up Flink parallelism while using exactly-once Kafka sinks requires careful management of Kafka transactional IDs; each parallel sink instance requires a unique transactional ID prefix configured via transactional.id.prefix
Exactly-once guarantees only apply within the Flink job and to the Kafka sink; any downstream consumer reading with isolation.level=read_uncommitted will see uncommitted (potentially aborted) messages
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp