Configure Kafka exactly-once semantics (EOS) for a transactional producer and idempotent pipeline

Steps

1. Enable idempotence on the producer: set enable.idempotence=true; this automatically sets acks=all, retries=Integer.MAX_VALUE, and max.in.flight.requests.per.connection=5
2. Assign a stable transactional.id to the producer (unique per logical producer instance, e.g. 'payments-producer-0'); this enables cross-session exactly-once guarantees
3. Initialize transactions in the producer application: producer.initTransactions() before sending any messages
4. Wrap each logical unit of work: beginTransaction() → send() → sendOffsetsToTransaction() (when consuming) → commitTransaction() or abortTransaction() on error
5. On the consumer side set isolation.level=read_committed so the consumer only reads messages from committed transactions and never sees aborted or in-flight transactional messages
6. Configure the broker-side transaction.state.log.replication.factor and transaction.state.log.min.isr to at least 2 for the internal __transaction_state topic to survive broker failures