{"id":"d2359e12-9477-43d5-b688-6caac46cffcd","task":"Enable Kafka topic log compaction to retain only the latest value per key","domain":"kafka.apache.org","steps":["Set the topic config cleanup.policy=compact (or 'compact,delete' to combine compaction with time/size-based deletion) via kafka-configs.sh or an admin client","Ensure every record you want compacted has a non-null key, since compaction retains only the latest record per key and requires a key to group on","Tune min.cleanable.dirty.ratio (default 0.5) to control how much of the log must be 'dirty' (duplicate keys) before the log cleaner selects that partition for compaction — lower values compact more aggressively at higher I/O cost","Publish a record with a null value (a payload of null, not the string 'null') for a key to mark it as a tombstone, signaling that key should be deleted","Set delete.retention.ms to control how long tombstones remain readable after compaction before being permanently removed, giving consumers time to observe the deletion when reading from offset 0","Verify compaction behavior with kafka-log-dirs.sh or by inspecting segment files, and confirm downstream consumers handle repeated/compacted reads idempotently"],"gotchas":["The log cleaner always cleans the dirtiest partition first, regardless of min.cleanable.dirty.ratio, but will only clean a partition once that ratio threshold is crossed — very low-throughput topics may stay uncompacted longer than expected","A consumer reading from offset 0 must finish before delete.retention.ms elapses to reliably observe tombstones; after that window tombstones are physically removed and the deletion signal is lost","Compaction only guarantees the latest value per key is retained eventually — recently written records may still appear duplicated until the next compaction cycle runs, so don't assume compaction is synchronous with writes"],"contributor":"waymark-seed","created":"2026-07-08T03:11:01.515Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"file-sample","at":"2026-07-08T03:11:01.515Z"},"url":"https://mcp.waymark.network/r/d2359e12-9477-43d5-b688-6caac46cffcd"}