Retries, idempotency, rotation

Webhooks are an at-least-once delivery channel. You will sometimes receive the same event_id twice. Build for it from day one.

Delivery state machine

Each delivery your endpoint receives went through this lifecycle on the producer side before it hit you:

pending → delivered   ✓ 2xx within 8 s
        ↘ failed      any non-retriable response (4xx other than 408/429, or 3xx) — first attempt
        ↘ pending → … → failed   408 / 429 / 5xx / timeout, after 6 attempts (~1 h 45 m)

What the producer guarantees:

• Idempotency on event_id. The same event_id may arrive multiple times, but always carries the same event_type and the same data payload. The producer never edits a previously-fired event.
• Per-source ordering inside a single event_type. Across event types, no ordering is guaranteed — subscription.upgraded for tenant A may arrive before payment.succeeded for tenant A even if the producer fired them in the opposite order.

What your handler must do

1. Look up event_id in your inbox table

   Maintain a table indexed by event_id with a unique constraint. On every request, attempt to insert a row before any business logic runs.

2. If the row already exists, return 200 immediately

   No further work needed. The producer counts that as delivered. Returning 200 on duplicates is correct — returning 4xx is wrong (drops the dedupe path you just gained).

3. Otherwise run your handler inside the same transaction as the insert

   Insert + business mutation as one atomic unit. If your handler throws, the insert rolls back too — the next retry has a clean slate.

4. Return 2xx (anything in 200..299) within 8 seconds

   204 No Content is fine. 200 {"received": true} is fine. The producer does not parse the body.

Backoff curve

Six attempts over roughly 1 h 45 m, on a fixed schedule with ±25 % jitter:

Attempt	Delay before this attempt	Cumulative since the first attempt
1	— (immediate)	0 s
2	30 s	30 s
3	2 min	2 min 30 s
4	10 min	12 min 30 s
5	30 min	42 min 30 s
6	1 h	1 h 42 min 30 s
failed	—	after attempt 6

After attempt 6 the delivery is marked failed and the subscription's owner is notified. The delivery log (GET /webhook_subscriptions/{id}/deliveries) shows every attempt with its response code and error, and any delivery can be re-sent from there — POST …/deliveries/{id}/replay mints a new delivery from the old payload, so your dedupe key still means "one attempt".

Why jitter

Every subscription that fails for the same reason at the same instant — a receiver restart, a deploy — would otherwise wake up on the same tick and thunder the receiver again. ±25 % spreads the wake-ups.

Retry-After honoured

If your endpoint responds with 429 Too Many Requests and includes a Retry-After header, we use that value instead of the next backoff step (capped at our longest slot, so a receiver cannot park a delivery for a day). Always return Retry-After: <seconds> when rate-limiting us — it lets you flatten our load spikes without us hammering.

Status codes you can return

Only codes that could plausibly answer differently on a replay are retried. Everything else is your verdict on this payload, so we stop rather than spending six attempts to be told the same thing — and the owner is alerted immediately, which is the signal they need to go fix the URL or the secret.

Status	What we do
200–299	delivered. No further attempts. The response body is not kept.
408, 429, 5xx, timeout, connection error	Retried on the schedule above.
410 Gone	Terminal and the subscription is paused — this is how a receiver deregisters itself. Re-arm it from Settings when you are ready.
Any other 4xx	Terminal on the first attempt. No retry.
3xx	Terminal — redirects are deliberately never followed (following one would let a public receiver bounce a delivery at an internal address).

Headers on retries

Every retry adds:

X-Kudos-Delivery-Attempt: 3
X-Kudos-Original-Event-Id: evt_…
X-Kudos-First-Sent-At: 2026-05-30T08:42:11Z

The event_id body / X-Kudos-Event-Id is stable across retries. Your inbox-lookup keys off it.

Replay events (operator-triggered)

A CRM operator can request a replay burst — Kudos CRM POSTs a sequence of historical events to a subscription. Replays are bracketed by control events:

1. subscription.replay_started — emit-side announces the replay. data.since, data.until, data.event_type filter.
2. Original events with the same event_id they had at first emission. Your inbox dedupes them away as duplicates.
3. subscription.replay_completed — emit-side closes out. data.events_emitted.

You do not have to do anything special during a replay — your idempotent handler from step 2 above is what makes replays safe.

Signing-secret rotation (overlap, not cutover)

Kudos CRM allows up to two active signing secrets per source concurrently. The contract is:

1. Operator clicks Rotate signing secret in the CRM console. Kudos CRM mints a new secret; the old one stays valid.
2. The operator pastes the new secret into your producer config alongside the old one. Your producer now emits the header with both signatures:

   X-Kudos-Signature: t=…,v1=<sig_with_old>,v1=<sig_with_new>

3. Kudos CRM receives the dual-signature header and verifies against any active secret on the source. Both succeed. No drop.
4. Once the operator confirms the new secret is in flight, they revoke the old one. Kudos CRM only verifies against the new secret from that moment on. Your producer can drop the old v1= from its emit list.

The same pattern applies in reverse for the reverse-channel signing secret.

If you ever see verification failures during a rotation, the most common cause is the producer dropped the old signature too early. Keep both signatures in the header until you have direct operator confirmation that the old secret was revoked.

Operational metrics worth tracking

Kudos CRM exposes 24-hour rolling stats per source in the operator console:

• total — count of events received
• delivered — count that processed cleanly
• discarded — count that returned a terminal 4xx (configuration / handler bugs)
• failed — count that exhausted the six attempts, or hit a terminal status code
• success_rate — delivered / total
• last_5xx — most recent 5xx timestamp

On your side we recommend tracking:

• inbox table size (controls dedupe lookup cost — vacuum old rows > 30 days)
• 95th-percentile handler latency (must stay under 8 s)
• handler error rate, segmented by event_type (lets you isolate one buggy mapper)

Move on to Bearer keys to set up the reconciliation pull channel.