Sign in
Skip to solution
hardAI Engineering

How do you evaluate a RAG pipeline end to end?

1.0k views
01

Understand the problem

Retrieval metrics (recall@k, MRR) plus generation metrics (faithfulness, relevance) — measured separately.

rag-evaluationrecallfaithfulnessevals
02

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Nudge consolestandby

Stuck? Beam a request up — the console returns a conceptual nudge that guides your logic without spoiling the implementation.

03

Study the solution

Evaluate the two stages separately. Retrieval: build a set of (query, relevant-chunk) labels and measure recall@k, precision@k, MRR/nDCG. Generation: given fixed retrieved context, judge faithfulness (is every claim supported by the context?), answer relevance and completeness — typically with an LLM judge validated ag

Solution ready — 2 min read

Classified // press E to declassify

04

Read the code

Retrieval eval: recall@k over a labeled set
def recall_at_k(cases, retriever, k=6):
    hits = 0
    for case in cases:                       # {"query": ..., "gold_chunk_ids": [...]}
        got = {c.id for c in retriever(case["query"], k=k)}
        if got & set(case["gold_chunk_ids"]):
            hits += 1
    return hits / len(cases)

# run on every index/chunking/reranker change:
# recall@6  baseline 0.71 → candidate 0.83   <- ship
# recall@6  baseline 0.71 → candidate 0.64   <- do not ship, whatever the demo showed
05

Join the discussion

Discussion (0)

Sign in to join the discussion.

No responses yet. Be the first to share what you think.

Transmission complete // awaiting log

KEEP THE
STREAK ALIVE.

Dossier 61 of 80 decoded in the AI Engineering track. One more won't hurt.

Back to track