Sign in
Skip to solution
hardAI Engineering

What techniques reduce LLM latency at each stage of a request?

241 views
01

Understand the problem

TTFT vs generation time: caching, smaller models, parallelism, and speculative tricks.

latencyttftperformanceoptimization
02

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Nudge consolestandby

Stuck? Beam a request up — the console returns a conceptual nudge that guides your logic without spoiling the implementation.

03

Study the solution

Split latency into time-to-first-token (prompt processing + queueing) and generation time (output tokens x per-token time). Reduce TTFT with prompt caching, shorter prompts and warm capacity; reduce generation time with smaller/faster models, capped or more concise outputs, and predicted-output/speculative techniques w

Solution ready — 2 min read

Classified // press E to declassify

04

Read the code

Parallelize independent LLM work
// sequential: ~3× the latency of the slowest call
// parallel: bounded by the slowest call only
const [summary, sentiment, entities] = await Promise.all([
  small.complete({ prompt: summarize(doc), maxTokens: 200 }),
  small.complete({ prompt: classifySentiment(doc), maxTokens: 5 }),
  small.complete({ prompt: extractEntities(doc), maxTokens: 300 }),
]);

// and route: only escalate the hard path to the frontier model
const answer = needsDeepReasoning(query)
  ? await frontier.complete({ prompt })
  : await small.complete({ prompt });
05

Join the discussion

Discussion (0)

Sign in to join the discussion.

No responses yet. Be the first to share what you think.

Transmission complete // awaiting log

KEEP THE
STREAK ALIVE.

Dossier 77 of 80 decoded in the AI Engineering track. One more won't hurt.

Back to track