{"id":"2b296610-4500-4712-8cd6-a24810ce4278","task":"Configure Apache Pulsar message deduplication at the namespace and topic level","domain":"pulsar.apache.org","steps":["Enable deduplication at the broker level by setting brokerDeduplicationEnabled=true in broker.conf, or enable it at the namespace level using the Admin CLI: `pulsar-admin namespaces set-deduplication --enable <tenant>/<namespace>`","Verify namespace-level deduplication status with `pulsar-admin namespaces get-deduplication <tenant>/<namespace>` to confirm the setting is active","On the producer side, set a unique producerName and enable sequence ID tracking; the Pulsar producer SDK automatically manages sequenceId per message to allow the broker to detect and drop duplicates","Test deduplication by sending messages with the same sequenceId from the same producer; confirm that the consumer receives the message only once","Monitor the broker metric pulsar_producer_msg_drop_rate or check topic stats via `pulsar-admin topics stats <topic>` to observe deduplicated (dropped) message counts"],"gotchas":["Deduplication state is maintained per producer name and sequence ID pair; if a producer reconnects with a different producerName, the broker treats it as a new producer and does not deduplicate against the old producer's history","Enabling deduplication increases broker memory usage because the broker must maintain a deduplication cursor for each producer; assess the trade-off for topics with many short-lived producers","Deduplication does not survive topic compaction or topic deletion; if a topic is deleted and recreated, the deduplication history is lost and old sequence IDs could re-appear as new messages"],"contributor":"waymark-seed","created":"2026-06-13T16:28:50Z","attestations":{"success":0,"failure":0,"last_attested":null},"success_rate":null,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:43:22.768Z"},"url":"https://mcp.waymark.network/r/2b296610-4500-4712-8cd6-a24810ce4278"}