When Retries Make Things Worse: Protecting Your System From Overload

Suppose a shipping quote provider starts taking two seconds to respond instead of 100 milliseconds. It still returns valid quotes. There are no authentication errors, no malformed responses, and no obvious reason to stop sending requests.

Our application waits. More customers reach checkout. Connections stay occupied, workers have less room for new requests, and some calls time out. The HTTP client retries those calls because a timeout looks temporary.

Now the provider receives more attempts while it has less capacity to finish them. Checkout gets slower, so we add application workers. They send even more requests to the same struggling provider.

Adding workers and retrying timeouts can turn this provider slowdown into an outage across our application, even though each change is meant to help checkout succeed.

Before adding either, I would ask whether repeating the operation is safe and whether the system can afford another attempt. In Idempotency: What, Why and How, the focus is the first question. Here, we need to answer the second.

Our shipping quote lookup is read-only. The examples use PHP, but the capacity problem applies to any application that waits for another service.

A request consumes resources while it waits. Depending on the runtime, those resources include a worker, a connection, memory for request state, or a slot in a connection pool. An asynchronous runtime avoids dedicating a thread to every wait, but the pending work still has a cost.

The following numbers illustrate the capacity problem. They are not production measurements:

Incoming quote requests:       100 per second
Average provider response:    0.1 seconds
Average concurrent calls:     about 10

Incoming quote requests:       100 per second
Average provider response:    2 seconds
Concurrent demand:            about 200

For a stable system, Little's Law relates average work in the system to arrival rate and average time spent there. The second calculation describes the concurrency we would need to sustain the same throughput at the slower response time. If our pool has only 50 slots, we cannot maintain that flow without waiting or rejecting work.

A queue starts to grow. Requests now spend time waiting for a connection before the provider even sees them. CPU usage in our application may remain low because the bottleneck is waiting on I/O. Scaling from CPU alone can miss the problem.

If a customer needs a quote within a second, spending most of that second waiting for a connection leaves little time for the provider to return a useful result.

An original request is one logical request for a quote. An attempt is one call to the provider, including the first call and any retries. Keep separate counters for them.

If 100 original requests arrive each second and every request reaches three attempts, the provider can receive 300 attempts per second. Three attempts means the initial call plus two retries.

If several layers retry independently, those allowances multiply:

Browser (3 total attempts)
    |
    v
Application (3 total attempts)
    |
    v
Provider client (3 total attempts)

3 x 3 x 3 = up to 27 provider attempts
for one original user action

That upper bound assumes each outer attempt repeats the downstream work and every inner layer exhausts its own allowance. Real traffic will not always reach the bound. The configuration still permits it when the dependency is least able to cope.

The AWS Builders' Library article on timeouts and retries describes this amplification and recommends choosing one retry point for many operations. Google's Handling Overload also explains why retry decisions need limits across requests, rather than only a maximum attempt count on each request.

For our quote lookup, I would make the provider adapter the retry owner. It knows which failures the provider considers temporary and whether another attempt is allowed. The surrounding application does not wrap the adapter in another retry loop. Any SDK or proxy retries must be included in that decision, or disabled for this call.

A timeout limits one wait. A deadline limits the time available to finish the whole operation.

Setting a 500 millisecond timeout on each of three attempts can spend 1.5 seconds waiting for calls, before backoff, connection acquisition, or application work. The customer may have stopped waiting long before the third attempt begins.

Instead, allocate time from the caller's remaining budget. Suppose checkout has 900 milliseconds left for quote lookup and response handling. We reserve 50 milliseconds for returning the result. Everything inside the lookup must share the remaining 850 milliseconds:

Quote lookup budget: 850 ms
┌────────────────────────────────────────────────────────────┐
│ First attempt │ Backoff │ Second attempt │ Remaining time   │
└────────────────────────────────────────────────────────────┘
0                                                        850 ms

The second attempt receives what remains.
It does not receive another 850 ms.

The actual values need latency measurements and load testing. An overly short timeout can manufacture failures by rejecting calls that would have completed normally. An overly long timeout keeps scarce resources occupied through an incident.

Use a monotonic clock for elapsed time inside a process. On 64-bit PHP, hrtime(true) returns an integer count of nanoseconds from an arbitrary origin. That origin is not a shared timestamp across servers, so do not send the raw value as a remote deadline.

For calls between services, propagate the remaining budget using the protocol's supported deadline mechanism. The gRPC deadline guide describes propagation and the need for application work to cooperate with cancellation. At a trust boundary, also cap any caller-supplied budget to the maximum our service accepts.

Check what the transport timeout actually covers. DNS, connection establishment, TLS, connection-pool waiting, and reading the response can have different controls, so a read timeout that looks sensible in the client configuration may still leave the whole call unbounded.

A concurrency limit caps how many attempts may be in flight at once. A rate limit caps how many attempts may start over time. Those controls solve different problems.

At 100 calls per second, a provider responding in 100 milliseconds needs much less concurrent capacity than the same provider responding in two seconds. A fixed request rate does not prevent that increase in concurrent work.

For the quote provider, put an admission check before connection acquisition. If there is no slot, return a controlled unavailable result or use a business-approved fallback. Do not silently move the wait into an unbounded semaphore queue.

The admission limit needs a clear scope. A limit of 10 calls in each of 20 replicas permits 200 calls across the fleet. Adding replicas increases that allowance unless we adjust the allocation. For a shared external quota, a gateway or coordinated allocation may be more appropriate.

Keep separate capacity for dependencies that should fail independently. Shipping quote waits should not consume the entire pool needed to read product information. This separation is often called bulkheading. It still requires enough capacity in the rest of the application to serve unrelated requests.

A client timing out does not prove that the provider stopped working. If we release a local slot and admit another call while the timed-out request still runs remotely, the provider may be doing more work than our local in-flight counter suggests.

Propagate cancellation where supported, measure remote completion where possible, and keep retries bounded even when cancellation is unavailable. Our client-side limit protects local waiting work. The provider also needs its own admission controls to enforce a remote execution limit.

A maximum of three attempts still allows every original request to triple its demand during an outage. A retry budget limits additional attempts across many original requests.

One option is to earn retry tokens as original requests arrive. For this example, each original request earns 0.1 token. Each retry spends one token, and the balance cannot exceed 20 tokens. Starting with an empty balance, 100 original requests earn at most 10 retries.

We can represent that allowance with integer credits:

final class RetryBudget
{
    private int $credits = 0;

    public function recordOriginal(): void
    {
        $this->credits = min(200, $this->credits + 1);
    }

    public function tryConsume(): bool
    {
        if ($this->credits < 10) {
            return false;
        }

        $this->credits -= 10;

        return true;
    }
}

Ten credits represent one token, which avoids floating-point accounting. The balance can hold enough credit for at most 20 retries, even after hours without a failure.

The object must outlive an individual request and be shared by the calls it governs. A new budget inside every lookup would start empty, earn only one credit, and never allow a retry. Sharing the object within one process gives that process a budget, not the whole fleet.

For concurrent threads or processes, the read-and-consume operation needs atomic storage or a retry mechanism that already provides that guarantee. The class above demonstrates the policy for one sequential owner. It is not a distributed counter. A gateway, a long-running client, or an SDK with documented retry quotas is often a better place to own this behaviour than application code.

Previously earned tokens allow a burst, so retries can exceed 10% of the original requests in a short observation window. This policy also leaves original traffic uncapped. Keep the admission limit even when the retry ratio looks low.

Backoff delays another attempt. Jitter spreads those delays so many clients do not retry at the same instant.

With full jitter, choose a random delay between zero and a growing cap. Our illustrative caps start at 25 milliseconds and double after a failed attempt, up to 200 milliseconds. A random delay still counts against the operation deadline.

If the provider sends a valid Retry-After, wait at least that long. If there is not enough time left to wait and make another useful attempt, stop. Shortening the requested delay to squeeze in a retry defeats the provider's overload signal.

Release the dependency slot before sleeping, then try to acquire it again before the next attempt. Holding that slot through backoff reduces the capacity available for other calls.

In a synchronous PHP request, usleep() still occupies an application worker. Releasing the dependency slot does not release that worker. Keep interactive retry delays short and bounded. Long recovery waits belong in a durable job or a later user action, with their own admission policy.

The retry owner needs a provider adapter that accepts a timeout and a limiter that can refuse work without waiting. These are application interfaces, not Laravel or package APIs:

interface QuoteClient
{
    public function fetch(string $postcode, int $timeoutMs): array;
}

interface ConcurrencyLimiter
{
    public function tryAcquire(): bool;

    public function release(): void;
}

final class TemporaryQuoteFailure extends RuntimeException
{
    public function __construct(
        public readonly ?int $retryAfterMs = null,
    ) {
        parent::__construct(
            'The quote provider is temporarily unavailable.',
        );
    }
}

QuoteClient::fetch() must enforce the supplied total attempt timeout across the transport work it owns and disable hidden retries. It translates only documented temporary failures into TemporaryQuoteFailure. Other errors propagate without another attempt. The adapter parses Retry-After into a non-negative duration when applicable.

The limiter must acquire a slot atomically without waiting. Its implementation belongs at the capacity scope we chose earlier. A per-request PHP object cannot enforce a fleet-wide limit.

With those contracts, the retry owner looks like this:

function fetchQuote(
    QuoteClient $client,
    ConcurrencyLimiter $limiter,
    RetryBudget $budget,
    string $postcode,
    int $remainingMs,
): array {
    $deadline = hrtime(true) + (min($remainingMs, 900) - 50) * 1_000_000;
    $budget->recordOriginal();

    for ($attempt = 1; $attempt <= 3; $attempt++) {
        $availableMs = (int) floor(($deadline - hrtime(true)) / 1_000_000);

        if ($availableMs < 50) {
            throw new RuntimeException('Quote deadline exhausted.');
        }

        if (! $limiter->tryAcquire()) {
            throw new RuntimeException('Quote capacity unavailable.');
        }

        try {
            if ($attempt > 1 && ! $budget->tryConsume()) {
                throw new RuntimeException('Quote retry budget exhausted.');
            }

            return $client->fetch($postcode, min(300, $availableMs));
        } catch (TemporaryQuoteFailure $failure) {
            if ($attempt === 3) {
                throw $failure;
            }
        } finally {
            $limiter->release();
        }

        $delayMs = random_int(0, min(200, 25 * 2 ** ($attempt - 1)));
        $delayMs = max($delayMs, $failure->retryAfterMs ?? 0);
        $availableMs = (int) floor(($deadline - hrtime(true)) / 1_000_000);

        if ($availableMs < $delayMs + 50) {
            throw $failure;
        }

        usleep($delayMs * 1_000);
    }

    throw new LogicException('The attempt loop must return or throw.');
}

The 50 millisecond minimum is an example policy for whether another attempt has enough time to be useful. The 300 millisecond attempt cap and 900 millisecond operation cap also need tuning from observed latency. These are not recommended defaults for every provider.

Only TemporaryQuoteFailure enters the retry path. Local capacity rejection and an exhausted retry budget return control to the caller without another provider attempt. A Retry-After longer than the remaining time also stops the loop. None of these decisions resets the operation deadline.

The snippet concentrates on retry decisions. A production boundary should represent deadline, capacity, and retry-budget exhaustion with distinct outcomes or exception types, then translate them to the endpoint's response contract. It also needs telemetry and cancellation support from the transport. The interfaces above do not supply those implementations.

This policy applies to a read-only quote lookup. A payment capture needs a documented idempotency contract before any timeout can be retried. A timeout tells us that the client lacks a response. It does not tell us that the provider performed no side effect.

Admission control deliberately leaves some work undone. We need to decide what the customer sees before an incident forces that choice.

For shipping quotes, possible policies include showing a clearly labelled estimate, offering a previous quote that is still valid for the same shipment, or asking the customer to retry later. The business has to approve the freshness and pricing rules. Inventing a zero-cost quote to keep checkout moving is a correctness bug.

For an HTTP endpoint, temporary service unavailability can be represented by 503. A caller-specific rate limit may use 429. Neither status is permission for immediate automatic retries. Define client behaviour, respect a supplied Retry-After, and avoid showing an endlessly spinning button that hides a permanent stream of new requests.

Rejecting excess work is load shedding. Backpressure means a producer responds to capacity signals by slowing production. An overload response provides no backpressure if the caller immediately sends the same request again.

A circuit breaker can stop calls after repeated failures and allow limited probes later. It can be useful when a dependency is consistently unavailable. I would first establish deadlines, admission limits, and retry accounting for this example. A circuit breaker introduces another stateful policy, and a poorly controlled recovery probe can restart the overload.

A retry test that throws once and succeeds on the second attempt proves very little about overload. It never creates competition for capacity.

Use a local provider stub and an arrival-rate load generator. For a controlled experiment, configure the stub with four execution slots, a 20 millisecond service time normally, and a 120 millisecond service time during the slowdown. Send 50 original requests per second.

At 120 milliseconds per call, four slots can theoretically finish about 33 calls per second, ignoring overhead. Our 50 original requests per second already exceed that capacity before a single retry. The experiment therefore creates overload even if the provider returns no errors of its own.

Run the same traffic profile against two configurations:

1. A baseline with up to three total attempts, a 300 millisecond attempt timeout, and no admission limit or shared retry budget.
2. The bounded policy, with four local dependency slots, the shared retry budget, jitter, and the same attempt cap. Give both configurations the same overall deadline so the comparison does not reward the bounded policy merely for waiting longer.

Start healthy, introduce the slowdown, and then restore normal service time without stopping original traffic. Run long enough to see whether queued work drains. Also run a healthy control with the protections enabled to check that they are not rejecting normal traffic.

The traffic generator must keep offering 50 original requests per second independently of response completion. A loop that waits for each response before sending another request reduces demand as the application slows down, which can hide the overload. Verify that the generator itself is not falling behind its schedule.

Measure original requests and provider attempts separately. Record successful quotes returned within the deadline, timeouts, local rejections, provider queue depth, and peak in-flight work. Measure successful-request latency separately from fast rejection latency, because a dashboard that combines both can make a system that rejects nearly everything appear to have excellent response times.

Check whether the bounded policy keeps admitted work within the configured limit and reduces retry amplification during the slowdown. Some customers will receive a controlled rejection. Measure successful throughput and recovery time on the actual stack before claiming an improvement. The parameters above describe a test to run, not benchmark results.

Repeat the experiment with a provider that ignores cancellation. Count work still running after the client timed out. That variant exposes the gap between limiting local waits and limiting remote execution, which a cooperative in-process fake would miss.

An error counter cannot tell us whether recovery behaviour is helping. For the quote operation, I would track these separately:

• Original request rate and provider attempt rate, including hidden transport attempts.
• Successful original requests completed within the operation deadline.
• Retries attempted, retries that recover a request, and retries denied by the budget.
• Current in-flight calls, admission rejections, and connection-pool wait time.
• Provider latency, attempt timeouts, and the age of queued work.

Divide provider attempts by original requests to see amplification over the same observation window. Remember that attempts can cross window boundaries. Inspect recovery after the slowdown, rather than expecting a single dashboard average to explain the whole incident.

Keep request IDs and trace IDs in traces or structured logs. Avoid using them as metric labels, where every new value creates another time series. Separate admission rejection from provider failure so an operator does not mistake a working protection for another provider error.

When a dependency slows down, I would resist increasing workers or retry counts until we know where the time and capacity are going. More callers cannot create capacity inside the provider.

I would accept some explicit quote rejections rather than let shipping requests consume the capacity needed by the rest of checkout. That choice needs a customer-facing response agreed in advance, not an improvised fallback during the outage.

Pick one dependency whose retries you cannot account for across the full call path. Count the attempts, make it slow in a controlled test, and keep sending traffic while it recovers.