Build a unified multi-carrier webhook ingestion service with dedupe, out-of-order event handling, and idempotency

Verified steps

1. Stand up a single HTTPS endpoint that accepts inbound webhooks from all integrated carriers; normalize each incoming payload into a canonical event schema (carrier, tracking_number, event_code, event_timestamp, location, raw_payload) before any downstream processing.
2. Immediately acknowledge every inbound webhook with HTTP 200 before doing any heavy processing — offload normalization and downstream work to an async queue (e.g., SQS, Pub/Sub, Kafka topic) to stay within carrier retry timeout windows.
3. Generate a deterministic idempotency key for each event, typically a hash of (carrier_id + tracking_number + event_code + event_timestamp); use this key as the upsert key in your event store to ensure re-deliveries of the same event are no-ops.
4. Store the event sequence number or timestamp alongside each persisted event; when applying events to a shipment's state machine, compare the incoming event's timestamp to the current state's timestamp and reject or queue events that would regress the state (e.g., an 'in transit' event arriving after a 'delivered' event).
5. Implement a resequencing buffer: hold out-of-order events in a pending queue for a configurable delay (e.g., 5-10 minutes) to allow lagging events to arrive before applying state transitions; after the window expires, apply in timestamp order and flag any irreversible out-of-order conditions.
6. Monitor the dead-letter queue for events that fail normalization or processing; alert on carrier-specific failure spikes which can indicate a breaking change in the carrier's webhook payload schema.