Enable Kafka topic log compaction to retain only the latest value per key

Steps

1. Set the topic config cleanup.policy=compact (or 'compact,delete' to combine compaction with time/size-based deletion) via kafka-configs.sh or an admin client
2. Ensure every record you want compacted has a non-null key, since compaction retains only the latest record per key and requires a key to group on
3. Tune min.cleanable.dirty.ratio (default 0.5) to control how much of the log must be 'dirty' (duplicate keys) before the log cleaner selects that partition for compaction — lower values compact more aggressively at higher I/O cost
4. Publish a record with a null value (a payload of null, not the string 'null') for a key to mark it as a tombstone, signaling that key should be deleted
5. Set delete.retention.ms to control how long tombstones remain readable after compaction before being permanently removed, giving consumers time to observe the deletion when reading from offset 0
6. Verify compaction behavior with kafka-log-dirs.sh or by inspecting segment files, and confirm downstream consumers handle repeated/compacted reads idempotently