Enable and configure Kafka tiered storage for a topic
domain: kafka · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Add a RemoteStorageManager plugin JAR to the broker classpath and configure remote.log.storage.system.enable=true and remote.log.storage.manager.class.name=<your-plugin-class> in server.properties
Configure the mandatory remote.log.metadata.manager.listener.name to the listener name the internal metadata manager client should use
Enable tiered storage on a specific topic: kafka-configs.sh --alter --topic my-topic --add-config 'remote.storage.enable=true,local.retention.ms=3600000,retention.ms=604800000'
Set local.retention.ms smaller than retention.ms so Kafka offloads segments older than local.retention.ms to remote storage while still retaining them remotely up to retention.ms
Optionally cap the remote-copy bandwidth per broker via remote.log.manager.copier.bytes.per.second in broker config to avoid saturating object-store egress
Verify offload progress by checking broker logs for RemoteLogManager and the remote storage metric group in JMX
Known gotchas
Kafka does not ship a built-in RemoteStorageManager; you must use a third-party plugin (e.g., from Aiven, Confluent, or your cloud provider) — there is no out-of-box S3 implementation in Apache Kafka itself
local.retention.ms and local.retention.bytes control when local segments become eligible for deletion after offload; setting them too low can cause local data to be deleted before the remote copy is confirmed, leading to data gaps on broker restart
Consumers reading tiered segments incur additional latency and cost because the broker fetches from remote storage on demand; use a fetch.max.wait.ms buffer or a dedicated fetch-cache tier for latency-sensitive consumers
Give your agent this knowledge — and 15,500+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?