Configure Pulsar topic compaction, retention, and TTL
domain: data-engineering · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Enable compaction on a topic by triggering it manually: pulsar-admin topics compact persistent://tenant/namespace/topic, or configure automatic compaction thresholds at the namespace level via namespace policies.
Set retention policy on a namespace (topics with no active subscriptions retain messages): pulsar-admin namespaces set-retention --size 10G --time 7d tenant/namespace.
Set a TTL so messages are automatically acknowledged and eligible for deletion after a time: pulsar-admin namespaces set-message-ttl --messageTTL 86400 tenant/namespace (value in seconds).
Backlog quota limits how much unconsumed data a topic can hold before applying a backlog policy (producer exception, consumer drop, oldest message drop): pulsar-admin namespaces set-backlog-quota --limit 5G --policy producer_request_hold tenant/namespace.
Verify compaction progress with pulsar-admin topics compaction-status persistent://tenant/namespace/topic.
Known gotchas
Compaction only retains the latest message per key; messages without keys are not compacted and all are retained.
Retention applies to messages that have been acknowledged by all subscriptions; unacknowledged messages (backlog) are subject to backlog quota, not retention.
TTL and retention interact: TTL causes messages to be auto-acknowledged, after which retention governs deletion; set both consistently.
Give your agent this knowledge — and 15,600+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?