PlaybookOct 9, 20267 min read

Retry or fix? A decision tree for every error in your logs

Half of all retry loops make incidents worse. Four questions decide whether to back off and try again, or stop and change the request.

Every page on KnowYourError carries a Retryable badge. This guide is the reasoning behind it, so you can apply it to errors we have not indexed yet.

1 · Did the server say "later"?

A 429, a 503, or a Retry-After header is an explicit invitation. Honour the header if present; otherwise use exponential backoff with full jitter.

retry.ts
async function withRetry(fn, { tries = 5, base = 200, cap = 8000 } = {}) {
  for (let i = 0; ; i++) {
    try { return await fn(); }
    catch (err) {
      if (i === tries - 1 || !isRetryable(err)) throw err;
      const wait = Math.random() * Math.min(cap, base * 2 ** i); // full jitter
      await new Promise(r => setTimeout(r, wait));
    }
  }
}
429Warning↻ retryableToo Many RequestsThe client sent more requests than the server's rate limit allows in the window. Wait Retry-After seconds, then retry with exponential backoff and jitter.

2 · Is the request idempotent?

A 504 means you do not know whether the work happened. Retrying a charge without an idempotency key can charge twice. Send the same Idempotency-Key on every attempt; Stripe keeps keys for 24 hours.

charge.ts
await stripe.paymentIntents.create(
  { amount: 4200, currency: "usd", customer },
  { idempotencyKey: `order_${order.id}` }
);

3 · Would the same input fail the same way?

Validation errors, 401, 403, 404, unique-constraint violations and card declines are deterministic. Retrying burns quota and, for payments, can trip fraud rules. Fix the payload or ask the user.

generic_declineWarningGeneric decline (card_declined)The issuer declined without giving Stripe a reason. Ask the customer to contact their bank or use another card; check Radar if many cards decline at once.

4 · Is someone else already retrying?

SDKs, queues and Lambda async invocations retry on their own. Stack your loop on top and three layers of 3 retries become 27 attempts.

Watch out

Lambda retries async invocations twice by default. A handler that retries 5 times internally can run 15 times per event.

  • Retry: 429, 503, 504 (with idempotency), Postgres 40P01, ECONNRESET.
  • Fix: 400, 401, 403, 404, 409, 422, P2002, card declines.
  • Investigate: 500 and 502. They can be either.