Your Retry Made It Worse

You post a letter and hear nothing back. Do you post a second one?

If the first letter never arrived, yes — obviously. If it arrived and the reply got lost, then posting again means the other side receives your request twice. And standing at the postbox, you cannot tell which of those two things happened.

That is the entire problem with retrying a failed request, and it is why the most obvious reliability measure in software is also one of the easiest to get exactly backwards.

Silence is not the same as failure

When a request fails with "could not connect," you know the work did not happen. Nobody was home.

When it fails with a timeout — meaning you gave up waiting — you know almost nothing. The request might never have arrived. It might have arrived and been refused. Or it might have arrived, run completely, saved to the database, charged the card, sent the email — and then the reply got lost coming back, or simply took longer than you were willing to wait.

From where you are standing, those look identical. That is not a shortcoming of your code. It is how networks are, and no amount of care on your side fixes it.

A timeout is a statement about your patience, not about what the other side did.

So the naive retry — catch the error, call again — is quietly making a claim it cannot back up. Most of the time the claim happens to be true and nobody notices. The rest of the time you get two of something that was meant to exist once.

Give each request a name tag

The fix is to stop guessing and let the other side tell you.

Before sending, you generate a unique ID for the operation — not for the attempt. Every retry of that same operation carries the same ID. The server writes that ID down alongside the result.

This is called an idempotency key. Idempotent is a fancy word for "doing it twice has the same effect as doing it once," like pressing a lift button.

def create_order(key: str, payload: dict) -> Order:
    existing = store.get(key)
    if existing:
        return existing          # this exact operation already ran

    order = process(payload)
    store.put(key, order)        # same transaction as process()
    return order

The important bit is that comment. Recording the key and doing the work must succeed or fail together. If the key is saved separately, there is a moment where the work is done and the key is not — and you have rebuilt the original bug with extra steps. Same database, same transaction, or the guarantee does not exist.

Now the ambiguity stops mattering. Retry as often as you like; the server either does the work or hands back what it did last time. The retry is safe not because it is careful, but because it is provably a no-op when it needs to be.

One thing this is not: it is not spotting duplicates by comparing the contents. Two genuinely separate orders for the same item at the same price are not duplicates. Only the caller knows whether two attempts mean the same thing — which is exactly why the caller generates the ID.

Reading can hurt too

Fetching data is naturally safe to repeat — reading a page twice changes nothing — so people retry reads without thinking.

That is fine for correctness and often terrible for survival, because the read you are hammering is usually the expensive one: the report, the big search, the summary. Retrying it three times during a slowdown triples the load on the exact thing that is already struggling.

Which brings us to the second half.

Retries pile on at the worst possible moment

Picture a service that is struggling and answering at half its normal speed.

Every caller times out. Every caller retries. Now twice as much work is arriving at something with half its usual capacity. It gets slower. More requests time out. More retries fire.

This is a retry storm, and it feeds itself: the system's own safety net is what stops it recovering.

Three things keep it in check, and you need all three.

Wait longer each time, with a random wobble. Backing off spreads the attempts out. The random part matters more than it sounds — without it, every caller in the fleet retries at the same instant, so you do not remove the stampede, you just make it rhythmic.

Cap retries as a share of all traffic. "Three retries per call" sounds strict and still permits tripling the load across a whole fleet. A budget — retries may not exceed 10% of requests — puts a ceiling on the amplification no matter how many callers are failing at once.

Stop calling entirely for a while. After enough failures in a row, give up and fail immediately for the next thirty seconds. This is the one that feels wrong, because you are deliberately not trying. It is also the one that lets the struggling service recover, because it removes the load holding it down. Engineers call it a circuit breaker, after the switch in your house that cuts the power before the wiring melts.

The AI version of all this

Every one of these problems is sharper in front of an AI model, because the calls are slow, expensive and often rate-limited.

Timeouts are common, so the window of doubt is wide. Calls cost real money, so a duplicate is not just wrong, it is a bill. Answers take many seconds, so retry storms build much deeper queues before anyone notices. And the most likely error you will hit is "too many requests" — precisely the one a naive retry loop turns into a sustained self-inflicted outage.

Here is the neat part. If a generated answer is expensive enough to be worth saving, it is expensive enough to deserve an idempotency key. Turn the question and its settings into an ID, store the answer against it, and a repeated request costs a lookup instead of another generation.

You get retry safety and a cache out of the same mechanism, which is a rare bargain.

The rule worth writing on the wall

Retrying is not error handling. It is a way of generating more load that happens to improve reliability when — and only when — three things are true: doing it twice is provably harmless, the error could actually succeed next time, and the total volume is capped.

Miss any one and you have not built a safety net. You have built an amplifier, and pointed it at the thing that was already in trouble.