Building Webhooks That Actually Deliver
A webhook that fires but doesn't land is worse than no webhook at all. Here's how we engineered Xtopay's delivery system to be reliable in low-connectivity environments.
Kofi Mensah
Lead Engineer at Xtopay
Webhooks are simple in concept: when something happens, we send an HTTP POST to your URL. But in practice — especially across African network infrastructure — the gap between 'sent' and 'delivered' is where most webhook systems fail. Intermittent connectivity, slow cold-start times for serverless endpoints, and undersized servers create a delivery reliability problem that standard retry logic doesn't fully solve.
What most webhook systems do wrong
Most webhook implementations send once, retry up to 5 times with fixed backoff, and then give up. If your server was restarting when the first attempt hit, or if your load balancer was shedding connections under traffic, you miss the event permanently. There's no recovery path.
Xtopay's delivery architecture
Durable queuing
Every webhook event is written to a durable queue before any delivery attempt begins. The queue persists events for 72 hours. If all delivery attempts fail, the event is still available — you can trigger a manual replay from the dashboard or API.
Exponential backoff with jitter
We retry on failure using exponential backoff: 5s, 30s, 2m, 10m, 30m, 2h, 6h, 24h. Jitter (±20%) prevents thundering herd on recovery. For the common case of a brief server restart, the 30-second retry catches the recovery without flooding your endpoint.
Across our current beta endpoints, 98.7% of webhook events are delivered successfully within 5 minutes of the triggering event. The remaining 1.3% resolve within 24 hours via our retry ladder.
Signature verification
Every delivery includes an X-Xtopay-Signature header — an HMAC-SHA256 signature of the payload using your webhook secret. Verify this on your end before processing any event.
- Retrieve your endpoint secret from the dashboard or GET /webhooks/{id}/secret
- Compute HMAC-SHA256(secret, raw_body)
- Compare to the value in X-Xtopay-Signature
- Reject any request where they don't match
Kofi Mensah
Lead Engineer at Xtopay
Building the high-fidelity payments infrastructure and cryptographic double-entry ledgers for emerging African business ecosystems.
