When a payment provider, ERP, CRM or shipping company reports an event via webhook, the endpoint that receives the request is the most fragile part of the integration. It needs to reject fake requests, not process the same event twice, not corrupt data with events arriving in reverse order and not keep the sending system waiting. Most webhook failures in the field come from skipping one of these four jobs: the signature is calculated over parsed JSON, business logic runs inside the request, no duplicate check is done.
Responsibilities of a webhook receiver
In corporate web and integration projects, a webhook receiver is an HTTP endpoint open to the internet, and its authentication relies not on a user session but on a shared secret. A correctly built receiver does these jobs, in this order:
- It reads the request’s raw body without modifying it and applies a size limit.
- It verifies the signature and timestamp; if they fail, it rejects the request before processing.
- It writes the event with its unique ID to a persistent inbox; if the ID has been seen before, it doesn’t write it again.
- It returns a 2xx to the sender quickly.
- It runs the actual business logic (updating the order status, transferring to the ERP) in a separate worker process.
The reason for this separation is simple. If the sender doesn’t get a response within a set time, it counts the request as failed and resends it. If business logic runs inside the request, a slow ERP call means both a timeout and the same event arriving again. The result: the same order is processed twice. The decision about which data to carry by webhook and which by queue or bulk transfer is covered in the ERP and web integration patterns guide; here we look only at the receiving endpoint.
HMAC signature verification and timestamp checks
In the common method, an HMAC-SHA256 digest of the request body is calculated with a secret shared by the sender and receiver and sent in a header. The HMAC construction is defined in RFC 2104. Providers differ in the header name and the signed content:
| Example | Signature header | Signed content |
|---|---|---|
| GitHub | X-Hub-Signature-256 (with a sha256= prefix) |
Raw body |
| Stripe | Stripe-Signature (t= and v1= fields) |
Timestamp + dot + raw body |
| Standard Webhooks | webhook-signature |
Event ID + timestamp + raw body |
If your own system sends webhooks to other systems, choose a scheme that includes the timestamp in the signature; for a clear description, the Standard Webhooks specification is a good starting point. If the timestamp isn’t signed, a captured valid request can be resent days later (a replay attack). The Python example below shows verification in a scheme that signs the timestamp; the header names are for illustration.
import hmac, hashlib, time
TOLERANCE_SEC = 300 # accepted clock difference
def is_signature_valid(raw_body: bytes, signature: str, timestamp: str, secret: bytes) -> bool:
try:
ts = int(timestamp)
except (TypeError, ValueError):
return False
if abs(time.time() - ts) > TOLERANCE_SEC:
return False # outside the replay window
signed = timestamp.encode() + b"." + raw_body
expected = hmac.new(secret, signed, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature) # constant-time comparison
Three details in this code are where things break most often in the field:
- The raw body must be used. Most web frameworks parse the JSON body automatically. A body that has been parsed and re-serialized doesn’t produce the same bytes because of differences in whitespace and field order; the signature doesn’t match. Do the verification using the framework’s method for accessing the raw body. This is the first checkpoint for the “right secret, wrong signature” complaint.
- The comparison must be constant-time.
==stops at the first differing character and creates a timing side channel; usehmac.compare_digestin Python andhash_equalsin PHP. - The server clock must be correct. The tolerance window assumes a clock synchronized with NTP. If the clock drifts by a few minutes, all requests are rejected. Look at the
timedatectloutput first.
During secret rotation, accept both secrets for a short transition period: first add the new secret to the receiver, then change it at the sender, and finally remove the old secret. If you reverse the order, events arriving during the switch get a 401. An IP allowlist can be an extra layer but doesn’t replace signature verification; cloud providers’ IP ranges change.
Fast acknowledgment, asynchronous processing
If the signature is valid, the receiver writes the event to a persistent table and responds immediately. The persistent write must complete before the response. An event dropped into an in-memory queue and answered with a 200 is lost if the process restarts. And the provider won’t send it again. An example inbox table for PostgreSQL:
CREATE TABLE webhook_inbox (
event_id text PRIMARY KEY,
source text NOT NULL,
event_type text NOT NULL,
received_at timestamptz NOT NULL DEFAULT now(),
payload jsonb NOT NULL,
status text NOT NULL DEFAULT 'pending',
attempts int NOT NULL DEFAULT 0,
last_error text
);
INSERT INTO webhook_inbox (event_id, source, event_type, payload)
VALUES ($1, $2, $3, $4)
ON CONFLICT (event_id) DO NOTHING;
Choose response codes deliberately. Return 401 or 403 for an invalid signature and 400 for a malformed body. If the write succeeds or the event already exists, return 200 or 202. If the database can’t be written to, a 5xx is right; the sender will retry. But don’t reflect a business logic error (for example, product not found in the ERP) back to the sender with a 5xx. Retrying won’t fix that error; it needlessly occupies the sender’s queue and with some providers goes as far as the endpoint being disabled. These errors are managed on the worker side with the status and last_error fields.
The worker process picks up pending records; in PostgreSQL, SELECT … FOR UPDATE SKIP LOCKED is the common way to prevent several workers from picking up the same record. After a set number of failed attempts, the record is moved to a separate status and goes to human review. The details of the retry policy are explained in the API error handling and retry guide.
Duplicate events and idempotent processing
Common webhook providers work with an “at-least-once” delivery model; they don’t guarantee exactly-once delivery. A network outage, a timeout or the receiver’s response not reaching the sender causes the same event to arrive more than once. Duplicates are normal. The defense has two layers:
- Filtering by event ID: The
event_idprimary key in the table above prevents the same event from being written a second time. If there’s no event ID, a hash key derived from the source, event type, source record ID and event time can be used. This is the second choice: it can collide if two genuinely different events occur in the same second. - Making the operation itself idempotent: Operations of the form “set stock to 37” instead of “reduce stock by 5”, and “add this payment if its ID doesn’t exist” instead of “add payment”, don’t change the result when repeated. Even if the event ID table is cleaned up one day, the second defense still works.
The inbox table shouldn’t grow without limit. Set a retention period longer than the provider’s longest retry period (for example a few weeks) and delete old records periodically.
Out-of-order events and a version field
The “order shipped” event can arrive before the “order confirmed” event. A receiver that applies events in order of arrival pulls the order back from “shipped” to “confirmed”; the customer dashboard goes backwards and the support line rings. There are three approaches:
- Conditional update with a version or sequence number: If the event carries the source record’s version or modification time, only a newer version is applied.
- Transition control with a state machine: Allowed transitions are defined; a transition from “shipped” to “confirmed” is rejected and the event is ignored.
- Thin event + re-read: The event carries only “this record changed”; the receiver reads the record’s current state from the source’s API. The ordering problem disappears; in return, every event costs an API call.
UPDATE orders
SET status = $1,
source_version = $2,
updated_at = now()
WHERE external_id = $3
AND source_version < $2; -- if an old version arrives, 0 rows are updated
If the number of rows updated is zero, the event is either old or the record doesn’t exist yet. To distinguish the second case, check whether the record exists. If it doesn’t, requeueing the event with a short delay is safer than silently dropping it.
Validation handshakes: the Microsoft Graph example
Some providers verify that the endpoint really belongs to you when the subscription is created. Microsoft Graph change notifications send the endpoint a request containing a validationToken query parameter when the subscription is created and expect this value to be returned as plain text within a short time. Each notification also carries the clientState value you set in the subscription; the receiver compares this value to verify that the notification belongs to its own subscription. A mistake we often see in the field is the receiver also putting this first request through the signature check, rejecting it, so the subscription is never created. How to narrow the application permissions on the Graph side is explained in the Microsoft Graph API permissions guide.
Testing, replay and monitoring
The receiver shouldn’t remain untested until the first event arrives in production. Before going live and on every significant change, go through this list:
- Is a validly signed request accepted, and rejected when a single byte in the body is changed?
- Is a timestamp outside the tolerance window rejected?
- When the same event is sent twice, is a single record created, and does the second request also get a 2xx?
- For two events sent in reverse order, is the final state correct?
- When the database is unreachable, does the receiver return a 5xx and does the provider retry?
- Are a large body limit and HTTPS-only access enforced?
- Is the feature to resend a past event (redelivery) from the provider’s panel known and tested?
In production, monitor at least these indicators: events received per minute, signature error rate, the age of the oldest event in pending status and the number of events that have fallen into failed status. The event count dropping to zero during expected hours is often a more important signal than a rise in the error rate. Silence is a failure too. How to set up this business-level monitoring is in the integration monitoring and reconciliation guide. For the scope of webhook- and API-based integration projects, take a look at our ERP, CRM and API integrations page.
Frequently asked questions
Is an IP allowlist enough for webhook verification?
No. An IP allowlist limits which network a request comes from; it doesn’t prove that the content hasn’t been altered or that the request isn’t being replayed. Providers’ IP ranges also change over time. The basic check is signature verification; an IP list, if there is one, remains only an additional layer.
Why does the signature sometimes not match despite the correct secret?
The most common reason is that the signature is calculated over JSON that the framework has parsed and re-serialized instead of over the raw body. When whitespace, field order or Unicode escaping changes, the digest changes too. Other reasons are the wrong character encoding, not stripping prefixes such as sha256= from the header, and server clock drift.
Which HTTP code should a webhook receiver return on error?
Return 401 or 403 for an invalid signature and 400 for a malformed request. If the event has been stored permanently, or was stored before, return 200 or 202. 5xx is used only for temporary infrastructure errors at the storage stage; business logic errors aren’t reflected back to the sender but managed on the receiving side.
How long should records be kept in the event ID table?
They should be kept longer than the longest retry period the provider documents. In practice, a few weeks of retention is enough in most scenarios. If the operations are also designed to be idempotent, a rare duplicate arriving after old records have been cleaned up won’t corrupt the data either.