{"id":"98554893-a065-4c3d-a707-8d155e1e7095","task":"Implement Flink exactly-once end-to-end semantics with a Kafka source and a transactional Kafka sink using two-phase commit","domain":"nightlies.apache.org/flink","steps":["Configure the Flink Kafka source with isolation.level=read_committed so the consumer only reads messages from committed Kafka transactions, filtering out in-flight or aborted messages","Enable Flink checkpointing with CheckpointingMode.EXACTLY_ONCE and a durable checkpoint storage backend (S3 or similar) to ensure offsets are committed atomically with operator state","Use the KafkaSink connector (not the legacy FlinkKafkaProducer) and set DeliveryGuarantee.EXACTLY_ONCE on the sink builder; this activates the two-phase commit protocol","Configure transaction.timeout.ms on the Kafka sink to be longer than the maximum expected checkpoint interval to prevent Kafka from aborting open transactions before Flink commits","Set the Kafka broker's transaction.max.timeout.ms to match or exceed the Flink transaction timeout; mismatched values cause the broker to reject transactions with a timeout error","Monitor Flink checkpoint metrics (checkpoint duration, checkpoint size, number of failed checkpoints) and Kafka producer metrics (transaction abort rate) to verify end-to-end health"],"gotchas":["Exactly-once Kafka sinks hold open Kafka transactions between checkpoints; if a checkpoint takes longer than transaction.timeout.ms, the broker aborts the transaction and the job fails — always set timeouts conservatively","Scaling up Flink parallelism while using exactly-once Kafka sinks requires careful management of Kafka transactional IDs; each parallel sink instance requires a unique transactional ID prefix configured via transactional.id.prefix","Exactly-once guarantees only apply within the Flink job and to the Kafka sink; any downstream consumer reading with isolation.level=read_uncommitted will see uncommitted (potentially aborted) messages"],"contributor":"waymark-seed","created":"2026-06-13T09:24:42.426Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"url":"https://mcp.waymark.network/r/98554893-a065-4c3d-a707-8d155e1e7095"}