mediumAI Engineering

What is prompt caching and how does it reduce cost and latency?

Q: What is prompt caching and how does it reduce cost and latency?

**TL;DR.** Providers cache the internal computation (**KV cache**) for a recently-seen **prompt prefix**; repeat calls sharing that prefix skip recomputation — cached input tokens cost up to ~90% less and **time-to-first-token drops sharply**. Exploit it by putting stable content first and keeping it **byte-identical**. call 1 system + tools + examples user msg A call 2 <text class='d-sub' x='190' y='99

599 views

Understand the problem

Reusing the KV-cache for stable prompt prefixes — the cheapest optimization most apps skip.

prompt-cachingcostlatencykv-cache

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Explicit cache breakpoints (Anthropic style)

await client.messages.create({
  model, max_tokens: 800,
  system: [
    { type: "text", text: BIG_STABLE_RUBRIC,          // 8k tokens, never changes
      cache_control: { type: "ephemeral" } },
  ],
  tools: TOOLS_SORTED_DETERMINISTICALLY,
  messages: [
    ...priorTurns,                       // growing cached prefix
    { role: "user", content: newQuestion },   // only this is fresh compute
  ],
});

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

What chunking strategies exist and how does chunk size affect RAG quality?

What is context engineering and how does it differ from prompt engineering?