hardAI Engineering

What techniques reduce LLM latency at each stage of a request?

Q: What techniques reduce LLM latency at each stage of a request?

**TL;DR.** Split latency into **TTFT** (queue + prompt processing) and **generation** (output tokens × per-token time). Cut TTFT with **prompt caching and shorter prompts**; cut generation with **smaller models and shorter outputs**; cut both at the app layer with **streaming, parallelism, precomputation and routing**. TTFT: queue + prefill generation: N output tokens × ms/token ↓ prompt caching ↓ shorter prompts</tex

241 views

Understand the problem

TTFT vs generation time: caching, smaller models, parallelism, and speculative tricks.

latencyttftperformanceoptimization

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Parallelize independent LLM work

// sequential: ~3× the latency of the slowest call
// parallel: bounded by the slowest call only
const [summary, sentiment, entities] = await Promise.all([
  small.complete({ prompt: summarize(doc), maxTokens: 200 }),
  small.complete({ prompt: classifySentiment(doc), maxTokens: 5 }),
  small.complete({ prompt: extractEntities(doc), maxTokens: 300 }),
]);

// and route: only escalate the hard path to the frontier model
const answer = needsDeepReasoning(query)
  ? await frontier.complete({ prompt })
  : await small.complete({ prompt });

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

How do you monitor LLM quality drift in production?

What is quantization and when would you use a quantized model?