Implement a reliable webhook processing pipeline with verification, deduplication, fast ack, and polling fallback

Verified steps

1. Verify the webhook signature before any processing: compute the expected signature from the raw request body bytes and your signing secret using the algorithm specified by the PSP (typically HMAC-SHA256); reject any request with a missing or non-matching signature with HTTP 400.
2. Respond with HTTP 200 (or 204) as fast as possible — within 5–10 seconds for most PSPs — before performing any database writes, downstream API calls, or business logic; use an async queue (job queue, message broker) to hand off processing.
3. Deduplicate by event ID: store each processed event ID in a database or cache with a TTL long enough to cover the PSP's retry window (24–72 hours); at the start of processing, check if the event ID has been seen and skip if so.
4. Implement idempotent handlers: every processing step (order fulfillment, email send, ledger entry) must be safe to run twice; use database upserts and unique constraints rather than blind inserts.
5. Set up a polling reconciliation job that runs on a schedule (e.g., every 15 minutes): query the PSP's list endpoint for recent events or payment statuses and compare against your local state to catch any webhooks that were never delivered.
6. Alert on dead-letter queue depth or reconciliation mismatches; a payment that is 'paid' in the PSP but 'pending' in your system for more than 30 minutes indicates a processing failure requiring investigation.