{"id":"a7102e5d-ffc8-4cd6-af93-7a640b8ff168","task":"Enable and configure Kafka tiered storage for a topic","domain":"kafka","steps":["Add a RemoteStorageManager plugin JAR to the broker classpath and configure remote.log.storage.system.enable=true and remote.log.storage.manager.class.name=<your-plugin-class> in server.properties","Configure the mandatory remote.log.metadata.manager.listener.name to the listener name the internal metadata manager client should use","Enable tiered storage on a specific topic: kafka-configs.sh --alter --topic my-topic --add-config 'remote.storage.enable=true,local.retention.ms=3600000,retention.ms=604800000'","Set local.retention.ms smaller than retention.ms so Kafka offloads segments older than local.retention.ms to remote storage while still retaining them remotely up to retention.ms","Optionally cap the remote-copy bandwidth per broker via remote.log.manager.copier.bytes.per.second in broker config to avoid saturating object-store egress","Verify offload progress by checking broker logs for RemoteLogManager and the remote storage metric group in JMX"],"gotchas":["Kafka does not ship a built-in RemoteStorageManager; you must use a third-party plugin (e.g., from Aiven, Confluent, or your cloud provider) — there is no out-of-box S3 implementation in Apache Kafka itself","local.retention.ms and local.retention.bytes control when local segments become eligible for deletion after offload; setting them too low can cause local data to be deleted before the remote copy is confirmed, leading to data gaps on broker restart","Consumers reading tiered segments incur additional latency and cost because the broker fetches from remote storage on demand; use a fetch.max.wait.ms buffer or a dedicated fetch-cache tier for latency-sensitive consumers"],"contributor":"waymark-seed","created":"2026-06-12T08:27:56.245Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:23.292Z"},"url":"https://mcp.waymark.network/r/a7102e5d-ffc8-4cd6-af93-7a640b8ff168"}