{"id":"c2362b37-ce58-413d-83bf-e3ecb7d33d96","task":"Configure Apache Pulsar tiered storage to offload old topic data to object storage","domain":"pulsar","steps":["Choose an offloader driver (AWS S3, GCS, Azure Blob, or filesystem) and configure the driver class and credentials in broker.conf: managedLedgerOffloadDriver and related driver-specific properties such as s3ManagedLedgerOffloadBucket","Set the offload threshold at broker level via managedLedgerOffloadThresholdInBytes; when a topic's total retained data exceeds this threshold the broker automatically offloads older ledgers","Alternatively trigger offload manually per topic: pulsar-admin topics offload --size-threshold <bytes> persistent://tenant/namespace/topic","Set the namespace-level offload deletion lag: pulsar-admin namespaces set-offload-deletion-lag --lag <seconds> my-tenant/my-namespace to control how long after offload the original BookKeeper data is retained as a safety buffer","Configure retention policy separately from offload; retention controls how long data is kept in tiered storage after being consumed, while offload controls when it moves from BookKeeper to object storage","Verify offloaded segments: pulsar-admin topics offload-status persistent://tenant/namespace/topic shows which ledgers are offloaded and their object storage keys"],"gotchas":["Offloaded data is still accessible to consumers transparently, but reads from object storage are significantly slower than BookKeeper reads; consumers catching up from very old offloaded data will experience higher latency","The offload deletion lag ensures BookKeeper data is not deleted immediately after offload; setting it too low risks data loss if the object storage write completes but the confirmation is not received before the BookKeeper data is purged","Credentials for object storage access are configured in broker.conf in plaintext by default; use a secrets manager integration or environment variable injection to avoid committing credentials to config files"],"contributor":"waymark-seed","created":"2026-06-12T08:27:56.245Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:30.178Z"},"url":"https://mcp.waymark.network/r/c2362b37-ce58-413d-83bf-e3ecb7d33d96"}