Implement streaming deduplication with keyed state and TTL in Flink or Kafka Streams

Steps

1. Choose a deduplication key (e.g., event_id, idempotency_key) that uniquely identifies a logical event.
2. In Flink SQL, use a ROW_NUMBER() window function partitioned by the dedup key and ordered by event_time, then filter WHERE row_num = 1 in a downstream view.
3. In Flink DataStream API, use a KeyedProcessFunction keyed on the dedup key; store a flag in ValueState<Boolean> and set a timer to clear it after the dedup window expires (state TTL).
4. Configure state TTL via StateTtlConfig.newBuilder(Time.hours(<n>)).setUpdateType(UpdateType.OnCreateAndWrite).build() to automatically purge state for keys not seen recently.
5. In Kafka Streams, use a persistent KeyValueStore to track seen IDs and a punctuator or TTL tombstone to expire old entries.
6. Test dedup effectiveness by replaying duplicate events and verifying exactly one output per logical event.