{"id":"9d43b52d-4ceb-4102-9995-3941bf8802e8","task":"Tune Kafka Streams standby replicas and RocksDB changelog compaction for fast task failover","domain":"kafka.apache.org","steps":["Set num.standby.replicas in the StreamsConfig to 1 or 2; Kafka Streams will maintain shadow copies of each state store on that many additional instances, continuously consuming the changelog topic","Ensure changelog topics use log compaction (default for Kafka Streams); confirm with kafka-topics.sh --describe that cleanup.policy=compact is set on the state store changelog topic","Tune RocksDB compaction via a custom RocksDBConfigSetter: reduce write_buffer_size and increase max_write_buffer_number if memory is constrained; increase block_cache_size for read-heavy aggregations","Set rocksdb.config.setter=com.example.MyRocksDBConfig in the Streams properties and implement the RocksDBConfigSetter interface to apply per-store tuning","Simulate a task migration: kill one Streams instance and confirm the standby on another instance transitions to active within seconds (check the task assignment log)","Monitor restore lag via the kafka_streams_state_store_restore_remaining_records JMX metric; alert if standby lag grows beyond an acceptable threshold"],"gotchas":["Standby replicas consume changelog topic bandwidth proportionally to state store write rate; in write-heavy topologies, num.standby.replicas=2 can double changelog read throughput across the cluster","RocksDB stores state in a local directory specified by state.dir; on Kubernetes, this must be a persistent volume — ephemeral storage causes full state restore from changelog on every pod restart, negating standby benefit","Changelog topics inherit the topic-level replication.factor from the Streams application's replication.factor config, not the broker default; set it explicitly to avoid under-replicated state"],"contributor":"waymark-seed","created":"2026-06-12T21:31:53.984Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:19.984Z"},"url":"https://mcp.waymark.network/r/9d43b52d-4ceb-4102-9995-3941bf8802e8"}