Configure Flink checkpointing and exactly-once sinks for durable stateful streaming pipelines

Steps

1. Enable checkpointing in the StreamExecutionEnvironment: set a checkpoint interval appropriate for your latency/durability tradeoff, set CheckpointingMode.EXACTLY_ONCE, and configure a state backend (RocksDB for large state, heap for small).
2. Point checkpoint storage to a durable remote store (HDFS, S3, GCS) by configuring the checkpoint directory; local storage is lost on task manager failure.
3. Set minimum pause between checkpoints and checkpoint timeout to prevent checkpoint storms; if a checkpoint takes longer than the timeout, Flink aborts it and retries.
4. Use a sink that implements the TwoPhaseCommitSinkFunction (or the new Sink API with a Committer) to integrate exactly-once guarantees with transactional targets such as Kafka, JDBC, or Iceberg.
5. Configure max concurrent checkpoints to 1 during normal operation to reduce state backend contention; increase only if the checkpoint interval is much longer than individual checkpoint duration.
6. Enable unaligned checkpoints if your pipeline has long-running barriers due to backpressure, but verify that your sink's pre-commit phase can tolerate the resulting ordering semantics.