Sign in
Skip to solution
mediumAI Engineering

How do you build resilient LLM calls with rate limits, retries and fallbacks?

1.2k views
01

Understand the problem

429s, overloaded errors and provider outages: the reliability layer every LLM app needs.

rate-limitsretriesfallbacksreliability
02

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Nudge consolestandby

Stuck? Beam a request up — the console returns a conceptual nudge that guides your logic without spoiling the implementation.

03

Study the solution

Wrap every model call in a resilience layer: retry transient failures (429/5xx/timeouts) with exponential backoff and jitter, respect rate-limit headers, and queue or shed load when sustained. For availability beyond one provider, define a fallback chain (same model other region, smaller same-family model, different pr

Solution ready — 2 min read

Classified // press E to declassify

04

Read the code

Resilience wrapper with fallback chain
const TIERS = [
  { name: "primary",  call: (r: Req) => anthropicUsEast(r) },
  { name: "region2",  call: (r: Req) => anthropicEuWest(r) },
  { name: "smaller",  call: (r: Req) => anthropicUsEast({ ...r, model: SMALL_MODEL }) },
];

async function completeResilient(req: Req) {
  for (const tier of TIERS) {
    if (breaker.isOpen(tier.name)) continue;              // fail fast past dead tiers
    try {
      return await retry(() => tier.call(req), {
        attempts: 3, backoff: expJitter(500), retryOn: [429, 500, 502, 503, "ETIMEDOUT"],
      });
    } catch (e) { breaker.record(tier.name, e); }
  }
  return degradedResponse(req);                            // cached answer + notice
}
05

Join the discussion

Discussion (0)

Sign in to join the discussion.

No responses yet. Be the first to share what you think.

Transmission complete // awaiting log

KEEP THE
STREAK ALIVE.

Dossier 16 of 80 decoded in the AI Engineering track. One more won't hurt.

Back to track