How One 429 Took Down an Agent Chain: Retry, Backoff, and Failover Postmortem

A 429 looks like a single-request problem. In an agent system, it can become a multiplier.

An agent may call a model to plan, invoke a tool, and call the model again to assemble the result. If every layer responds to 429 with “try again immediately,” one user request becomes dozens of upstream attempts. Those attempts keep the rate-limit window saturated while new agent steps add more traffic. The outcome is a retry storm.

This post does more than repeat “use exponential backoff.” It provides:

1. a synthetic multi-step agent incident timeline;
2. a Python standard-library replay of three retry strategies;
3. an auditable state machine for classifying 429, waiting, switching channels, and enforcing a total budget.

Every request count, recovery time, and result comes from a fixed synthetic fixture. No live model API was called. The results do not represent any provider's actual limits, availability, or performance.

How the failure multiplies

Assume one request runs this chain:

user request
  → planning model
  → search tool
  → synthesis model
  → final answer

If each stage allows up to three attempts, the worst case is not three attempts:

3 × 3 × 3 = 27 model or tool calls

That excludes retries by:

• the client SDK;
• the LLM gateway;
• the job queue;
• the user's Retry button;
• parallel agent branches.

Synthetic incident timeline

This timeline demonstrates the mechanism. It is not a production incident record.

Time	Event	Bad action	Immediate effect
00:00	120 agent subrequests arrive	all target one channel	upstream begins returning 429
00:00–00:10	first 429 wave	clients retry immediately	traffic doubles inside one window
00:10–00:500	second and third waves	SDK and gateway both retry	the rate bucket stays saturated
00:500–00:1750	synchronized exponential backoff	every client uses the same delay	each wake-up creates another spike
00:2000	upstream recovers	jittered attempts begin succeeding	part of the agent fleet completes
00:2500	total budget expires	remaining retries stop	tail latency becomes bounded

The important question is not simply whether 429 should be handled. It is which layer owns retries, how many attempts it may make, what deadline it shares, and whether another layer is retrying the same operation.

429 is not one error

OpenAI's current error guide distinguishes at least two common 429 meanings:

• requests are arriving too quickly;
• credits, quota, or monthly spend are exhausted.

The first may recover after waiting. The second is usually not fixed by sleeping for a few hundred milliseconds. Anthropic documents separate request, input-token, and output-token limits and exposes limit, remaining, and reset data in response headers. Google's 429 guidance also distinguishes pay-as-you-go capacity pressure from Provisioned Throughput behavior.

Do not reduce the policy to:

if status == 429:
    retry()

Classify first:

Result	Retry by default?	Reason
transient RPM/TPM/concurrency pressure	yes, after a bounded wait	a window may recover
explicit Retry-After	wait as instructed	do not override server pacing
account quota or budget exhausted	no	waiting does not add quota
body or context limit exceeded	no, not unchanged	input or model must change
invalid request, authentication, permission	no	retries cannot repair configuration
safety refusal or unsupported capability	no blind cross-model retry	it may change semantics or expand risk

The HTTP status is only one signal. A decision also needs error type, response headers, account budget, idempotency, and the remaining deadline.

Offline replay of three strategies

The fixture is fixed:

• 120 clients start at t=0;
• the upstream recovers at 2,000ms;
• each client gets at most four attempts;
• collisions use 100ms buckets;
• the total retry budget is 2,500ms;
• random seed 107 makes the result reproducible.

Only requests at t >= 2,000ms succeed. This does not emulate a provider's internal algorithm. It compares the timing shape of three schedulers.

Strategy A: immediate retry

delay = 0

Result:

• requests: 480
• completed: 0
• completion rate: 0%
• peak inside the throttled window: 480
• final attempt: 0ms

The 120 first attempts and all three retries happen at the same instant. The policy converts one 429 wave into a larger 429 wave.

Strategy B: exponential backoff without jitter

delay = 250 × 2^(attempt - 1)

Result:

• requests: 480
• completed: 0
• completion rate: 0%
• peak inside the throttled window: 120
• final attempt: 1,750ms

The instantaneous peak is lower, but every client wakes at 250ms, 750ms, and 1,750ms. Synchronized backoff turns one spike into several orderly spikes.

Strategy C: exponential backoff, jitter, and total budget

base = 250 × 2^(attempt - 1)
delay = min(base + random(0, 250), 1500)

A retry is discarded when its scheduled time would exceed the total budget.

Result:

• requests: 480
• completed: 94
• completion rate: 78.3%
• peak inside the throttled window: 120
• final attempt: 2,438ms

This result does not prove that these delay constants are optimal in production. It demonstrates three reproducible properties:

1. immediate retry pins every request to one instant;
2. exponential backoff without jitter preserves client synchronization;
3. jitter plus a total time budget lets some requests reach the recovery window without unbounded tail latency.

The initial collision is still 120 because all 120 first attempts arrive together. A production system also needs admission control, queues, token buckets, or concurrency limits. Retry policy cannot repair an already overloaded ingress by itself.

Core of the replay

def next_delay(attempt, rng):
    base = 250 * (2 ** (attempt - 1))
    return min(base + rng.randrange(0, 251), 1500)

next_at = current_at + next_delay(attempt, rng)
if next_at <= total_budget_ms:
    schedule(next_at)

The script uses only the Python standard library and makes no network request. The table above is real local execution output, not mental arithmetic.

A safe state machine: classify, wait, then switch

A gateway or agent client needs at least these states:

START
  → SEND
  → SUCCESS
  → CLASSIFY_ERROR
       ├─ invalid/auth/permission/safety → FAIL
       ├─ quota/budget exhausted         → FAIL + alert
       ├─ transient 429/5xx/timeout      → WAIT
       └─ stream already started         → STOP transparent replay
  → WAIT_WITH_JITTER
  → CHECK_DEADLINE_AND_BUDGET
       ├─ expired → FAIL
       └─ available → RETRY_SAME_CAPABILITY
  → if deployment unhealthy → FALLBACK_COMPATIBLE_CHANNEL
  → if no safe candidate → FAIL

Why the first fallback should preserve model capability

When the primary channel is temporarily throttled, the lowest-risk alternative is usually:

• the same model;
• equivalent tool support;
• the same context and output guarantees;
• the same region and retention policy;
• another healthy deployment or API key.

Cross-model fallback should happen only when the candidate passes the same hard constraints and the product permits a semantic downgrade. “The cheap model is available” does not mean it is compatible.

LiteLLM's first-party documentation separates fallback after num_retries from balancing deployments under the same model group. That boundary is useful: retry count, cooldown, and cross-model fallback should be recorded separately.

Streaming adds one red line

After the first visible token reaches the client, a gateway cannot safely replay the whole request from the beginning:

• the client may already display partial text;
• tool arguments may be only half assembled;
• repeating a tool can duplicate side effects;
• two generations create duplicate content and ambiguous billing.

A baseline policy is:

• failure before first token: retry or switch a compatible channel within the deadline;
• disconnect after first token: record a partial stream and return a protocol-defined recoverable state;
• tool already executed: require an idempotency key, transaction, or human confirmation; never transparently replay side effects.

What Nbility currently does

In the synchronized /opt/data/nbility code, the main relay loop currently:

1. caps attempts with common.RetryTimes;
2. selects a channel by group, model, request path, and usage state on each attempt;
3. uses priority tiers and weight within a tier;
4. evaluates continuation through shouldRetry and shouldRetryTaskRelay;
5. supports configurable HTTP status-code retry ranges;
6. explicitly excludes special cases such as 504, 524, and bad response bodies;
7. records use_channel and error details and may disable channels according to policy;
8. keeps an asynchronous task pending when polling receives 429 instead of immediately marking it failed.

The default HTTP retry range is not “429 only.” Current code includes multiple 1xx, 3xx, selected 4xx, and selected 5xx statuses while excluding 504 and 524. Production operators should tune those rules to upstream semantics and business idempotency instead of assuming that the presence of retry implies complete exponential backoff and jitter.

Boundary: the current main relay loop demonstrates attempt counts, channel selection, status filtering, and error records. The exponential backoff, jitter, total deadline, and admission control used in this article's simulation must not be presented as existing Nbility features.

Plausible-looking mistakes

Mistake 1: every layer retries

The client retries once, the SDK twice, the gateway three times, and the agent step three times. Attempts may combine multiplicatively, not as 1 + 2 + 3 + 3.

Define one retry owner. Other layers should propagate the error and remaining deadline. Traces should record retry attempt, owner, and parent request ID.

Mistake 2: every 429 means temporary congestion

Exhausted credit, account budget, unsupported models, and request-policy limits do not become healthy after a short sleep.

Mistake 3: cross-model fallback ignores capability

A vision request cannot silently fall back to a text-only model. A strict-schema tool call is not compatible merely because the backup returns HTTP 200.

Mistake 4: attempts are capped but time is not

“Three retries” is not a deadline. Long backoffs can hold user and agent concurrency for tens of seconds. Before every attempt, check:

remaining_deadline > estimated_backoff + minimum_attempt_time

Mistake 5: replaying a stream that already started

This creates duplicate text, repeated tool calls, and bills that are difficult to reconcile. Track whether any visible token has been emitted.

Production checklist

• assign one retry owner
• distinguish RPM, TPM, concurrency, credit, and budget failures
• prefer provider Retry-After and rate-limit headers
• use exponential backoff with jitter
• enforce both maximum attempts and a total deadline
• limit ingress concurrency before overload reaches the provider
• retry only recoverable failures
• never retry authentication, permission, invalid input, or safety refusal unchanged
• never transparently replay the full stream after visible output
• validate model capability, region, tools, and schema before fallback
• protect tool side effects with idempotency or transactions
• record retry owner, attempt, deadline, channel, and final state
• use cooldown or a circuit breaker to avoid unhealthy channels
• replay fixed fixtures before shadow traffic and a small canary

Conclusion

A 429 incident usually exposes missing retry ownership and failure boundaries, not simply insufficient upstream capacity.

The reliable sequence is:

1. classify the exact 429 meaning;
2. control admission so new traffic does not deepen overload;
3. assign one retry owner;
4. respect Retry-After and add jittered backoff;
5. check total budget and idempotency;
6. prefer a healthy channel with the same model, capability, and compliance boundary;
7. degrade across models only when explicitly allowed;
8. fail clearly when a stream has started or no safe candidate remains.

Nbility already provides a unified endpoint, channel priorities and weights, configurable status retry rules, channel records, and billing boundaries. A complete retry budget, jitter, circuit breaker, and agent-level trace still require implementation and validation in the gateway or caller. Visit Nbility for unified model access, or request a small test credit through the ticket center.

Official and first-party sources

• OpenAI Rate Limits
• OpenAI Error Codes
• Anthropic Rate Limits
• Anthropic Errors
• Google Cloud Gemini Error Code 429
• Google Cloud Retry Strategy
• AWS SDK Retry Behavior
• LiteLLM Provider Failover