Sign in
Skip to solution
hardAI Engineering

What is semantic caching and when is it safe to use?

954 views
01

Understand the problem

Serving cached answers for similar (not identical) queries — big savings, real correctness risks.

semantic-cachingcostcachingrisk
02

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Nudge consolestandby

Stuck? Beam a request up — the console returns a conceptual nudge that guides your logic without spoiling the implementation.

03

Study the solution

Semantic caching embeds incoming queries and, if a previous query is within a similarity threshold, returns its cached answer instead of calling the model. It can eliminate a large share of LLM calls in FAQ-like traffic, but false positives are dangerous: similar wording does not guarantee identical intent ('cancel my

Solution ready — 2 min read

Classified // press E to declassify

04

Read the code

Guarded semantic cache lookup
async function cachedAnswer(q: string, ctx: Ctx) {
  if (TRANSACTIONAL.test(q)) return llm(q, ctx);          // never cache these

  const key = { tenant: ctx.tenantId, promptV: PROMPT_VERSION, model: MODEL };
  const hit = await semCache.nearest(await embed(q), { where: key, minSim: 0.95 });
  if (hit && !expired(hit, ttlFor(hit.topic))) {
    metrics.increment("semcache.hit");
    return hit.answer;
  }
  const ans = await llm(q, ctx);
  await semCache.put({ ...key, query: q, answer: ans });
  return ans;
}
05

Join the discussion

Discussion (0)

Sign in to join the discussion.

No responses yet. Be the first to share what you think.

Transmission complete // awaiting log

KEEP THE
STREAK ALIVE.

Dossier 63 of 80 decoded in the AI Engineering track. One more won't hurt.

Back to track