Describe in the specification how an integration behaves on failures: the timeout, which errors to retry and with what delays, the idempotency key, discarding duplicates, the dead letter queue and the outbox.
When there is no response
When there is no response
A timeout doesn't mean failure: the operation may have succeeded, with the money charged and only the response lost. An unknown outcome is the main source of duplicates.
Retry only outcomes that may succeed next time (table below). Limit retries by:
• the number of attempts, e.g. 3;
• doubling delays (1 s, 2 s, 4 s…) plus a random shift (jitter), so thousands of clients don't hit at once;
• a total time budget for all attempts.
Idempotency key. Only idempotent calls are safe to retry; POST /orders isn't one. So the client generates a unique Idempotency-Key (UUID) per checkout attempt and sends it with every retry. The server stores the key with the result (say, 24 hours) and returns the same response to a retry. The same key with a different body is an error, usually 422. It's common practice (Stripe and others) and an IETF draft, not an RFC.
The consumer discards duplicates (Idempotent Receiver): it stores processed eventIds in the processed_events table and skips repeats. A message that fails N times or is malformed goes to the dead letter queue (DLQ) so it doesn't block the rest; the DLQ is monitored and reviewed.
Dual write: the order is in the database, but the event never reached the broker (or vice versa). The fix is a transactional outbox: the event goes into the outbox table in the same transaction as the order, and a separate process publishes it at least once.
Outcome → action
| Outcome | Retry? | How |
|---|---|---|
| timeout, dropped connection | yes | only if the operation is idempotent or carries a key |
| 502 / 503 / 504 | yes | with a growing delay and jitter |
| 429 | yes | after the delay given in Retry-After |
| 400 / 422 | no | fix the request; a malformed message goes to the DLQ |
| 401 / 403 / 404 | no | without a fix, the response will be the same |
| 409 | no | re-read the state of the resource |
Lesson notes
When there is no response
A timeout doesn't mean failure: the operation may have succeeded, with the money charged and only the response lost. An unknown outcome is the main source of duplicates.
Retry only outcomes that may succeed next time (table below). Limit retries by:
• the number of attempts, e.g. 3;
• doubling delays (1 s, 2 s, 4 s…) plus a random shift (jitter), so thousands of clients don't hit at once;
• a total time budget for all attempts.
Idempotency key. Only idempotent calls are safe to retry; POST /orders isn't one. So the client generates a unique Idempotency-Key (UUID) per checkout attempt and sends it with every retry. The server stores the key with the result (say, 24 hours) and returns the same response to a retry. The same key with a different body is an error, usually 422. It's common practice (Stripe and others) and an IETF draft, not an RFC.
The consumer discards duplicates (Idempotent Receiver): it stores processed eventIds in the processed_events table and skips repeats. A message that fails N times or is malformed goes to the dead letter queue (DLQ) so it doesn't block the rest; the DLQ is monitored and reviewed.
Dual write: the order is in the database, but the event never reached the broker (or vice versa). The fix is a transactional outbox: the event goes into the outbox table in the same transaction as the order, and a separate process publishes it at least once.