mediumAI Engineering

How do you build resilient LLM calls with rate limits, retries and fallbacks?

Q: How do you build resilient LLM calls with rate limits, retries and fallbacks?

**TL;DR.** Wrap every model call in a resilience layer: **exponential backoff + jitter** on 429/5xx/timeouts honoring Retry-After , **queue or shed** sustained overload, a **fallback chain** (other region → smaller sibling → other provider) with the honesty that outputs differ, **circuit breakers** to fail fast, and **explicit degraded modes** in the product. request primary <text class='d-sub' x='186' y='90' text-anchor='mi

1.2k views

Understand the problem

429s, overloaded errors and provider outages: the reliability layer every LLM app needs.

rate-limitsretriesfallbacksreliability

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Resilience wrapper with fallback chain

const TIERS = [
  { name: "primary",  call: (r: Req) => anthropicUsEast(r) },
  { name: "region2",  call: (r: Req) => anthropicEuWest(r) },
  { name: "smaller",  call: (r: Req) => anthropicUsEast({ ...r, model: SMALL_MODEL }) },
];

async function completeResilient(req: Req) {
  for (const tier of TIERS) {
    if (breaker.isOpen(tier.name)) continue;              // fail fast past dead tiers
    try {
      return await retry(() => tier.call(req), {
        attempts: 3, backoff: expJitter(500), retryOn: [429, 500, 502, 503, "ETIMEDOUT"],
      });
    } catch (e) { breaker.record(tier.name, e); }
  }
  return degradedResponse(req);                            // cached answer + notice
}

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

When should you use batch APIs instead of real-time LLM calls?

How does streaming work in LLM APIs and why does it matter for UX?