Steps

Choose an offloader driver (AWS S3, GCS, Azure Blob, or filesystem) and configure the driver class and credentials in broker.conf: managedLedgerOffloadDriver and related driver-specific properties such as s3ManagedLedgerOffloadBucket
Set the offload threshold at broker level via managedLedgerOffloadThresholdInBytes; when a topic's total retained data exceeds this threshold the broker automatically offloads older ledgers
Alternatively trigger offload manually per topic: pulsar-admin topics offload --size-threshold <bytes> persistent://tenant/namespace/topic
Set the namespace-level offload deletion lag: pulsar-admin namespaces set-offload-deletion-lag --lag <seconds> my-tenant/my-namespace to control how long after offload the original BookKeeper data is retained as a safety buffer
Configure retention policy separately from offload; retention controls how long data is kept in tiered storage after being consumed, while offload controls when it moves from BookKeeper to object storage
Verify offloaded segments: pulsar-admin topics offload-status persistent://tenant/namespace/topic shows which ledgers are offloaded and their object storage keys

Known gotchas

Offloaded data is still accessible to consumers transparently, but reads from object storage are significantly slower than BookKeeper reads; consumers catching up from very old offloaded data will experience higher latency
The offload deletion lag ensures BookKeeper data is not deleted immediately after offload; setting it too low risks data loss if the object storage write completes but the confirmation is not received before the BookKeeper data is purged
Credentials for object storage access are configured in broker.conf in plaintext by default; use a secrets manager integration or environment variable injection to avoid committing credentials to config files

pulsar.apache.org · 5 steps · unrated

Configure Apache Pulsar message deduplication at the namespace and topic level

pulsar.apache.org · 5 steps · unrated

Configure Pulsar topic compaction, retention, and TTL

data-engineering · 5 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Configure Apache Pulsar tiered storage to offload old topic data to object storage

Steps

Known gotchas

Related routes

Give your agent this knowledge — and 15,500+ more routes

Need this verified for your stack — or a route we don't have yet?