Stop Writing Clever Retry Loops: Backoff, Jitter and Knowing When to Quit

Last quarter a payment provider we depend on had an eleven-minute hiccup. Our service survived it. Our retry logic, however, turned eleven minutes into forty, because every instance hammered the provider in perfect lockstep the moment it came back.

Retries feel like free resilience. They are not. They are load you add to a system that is already struggling.

The three mistakes I keep seeing

• Fixed delays. Sleeping 1 second between attempts means 500 pods all retry at the same moment. That is a self-inflicted traffic spike.
• Retrying everything. A 400 Bad Request will be just as bad on the fifth try. Only retry things that can plausibly succeed later: timeouts, 503, connection resets.
• No ceiling. I have seen loops with while (true). I have seen them in production. I have seen them at 3am.

The boring version

Here is roughly what we standardised on. It fits on one screen.

async function withRetry(fn, max = 4, base = 200) {
  for (let attempt = 0; ; attempt++) {
    try { return await fn() }
    catch (err) {
      if (!isRetryable(err) || attempt >= max) throw err
      const cap = base * 2 ** attempt
      await sleep(Math.random() * cap)
    }
  }
}

Two details matter. The delay grows exponentially, so pressure on the dependency drops fast. And the actual sleep is a random value between zero and that cap, which is called full jitter. Jitter is what breaks the lockstep. After adding it, our retry spike during a test outage went from a sharp wall to a gentle hill.

Know when to quit

Four attempts with a 200ms base means the worst case waits around three seconds total. That is our budget for a user-facing request. For a background job we allow more attempts, but we also put the job back on the queue instead of holding a worker hostage.

We also added a simple rule: if more than half the calls to a dependency failed in the last 30 seconds, stop retrying entirely and fail fast. Users get a clear error. The dependency gets room to recover.

Retries need idempotency

If you retry a charge, you must be sure the first one did not quietly succeed. We send an idempotency key with every write, generated once per logical operation, not per attempt. Without it, retries are a polite way of charging customers twice.

A checklist for your next PR

1. Is the error actually retryable?
2. Is there exponential backoff with jitter?
3. Is there a hard cap on attempts and total time?
4. Is the operation idempotent?
5. Is there a log line when we give up?

If all five answers are yes, the retry loop can be boring. Boring is exactly what you want during an outage.

Cover: Photo by Scott Rodgerson on Unsplash.