hardAI Engineering

What is semantic caching and when is it safe to use?

Q: What is semantic caching and when is it safe to use?

**TL;DR.** Semantic caching **embeds incoming queries** and returns a **previous answer** when a past query is within a similarity threshold. Huge savings on FAQ-like traffic — but similar wording ≠ same intent, so it is only safe for **idempotent, non-personalized, informational** queries with high thresholds, tenant scoping and TTLs. query embed similarity ≥ 0.95?

954 views

Understand the problem

Serving cached answers for similar (not identical) queries — big savings, real correctness risks.

semantic-cachingcostcachingrisk

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Guarded semantic cache lookup

async function cachedAnswer(q: string, ctx: Ctx) {
  if (TRANSACTIONAL.test(q)) return llm(q, ctx);          // never cache these

  const key = { tenant: ctx.tenantId, promptV: PROMPT_VERSION, model: MODEL };
  const hit = await semCache.nearest(await embed(q), { where: key, minSim: 0.95 });
  if (hit && !expired(hit, ttlFor(hit.topic))) {
    metrics.increment("semcache.hit");
    return hit.answer;
  }
  const ans = await llm(q, ctx);
  await semCache.put({ ...key, query: q, answer: ans });
  return ans;
}

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

How do you evaluate a RAG pipeline end to end?

When should you split work across multiple agents instead of one?