hardAI Engineering

How do you evaluate a RAG pipeline end to end?

Q: How do you evaluate a RAG pipeline end to end?

**TL;DR.** Evaluate the **two stages separately**. Retrieval: labeled (query → relevant chunk) pairs scored with **recall@k, precision@k, MRR/nDCG**. Generation: with retrieval held fixed, judge **faithfulness, relevance, completeness** (LLM judge validated against humans). End-to-end accuracy alone cannot tell a retrieval miss from a hallucinating generator. stage 1: retrieval labels: query → gold chunks recall@k · precision@k MRR · nDCG <rect class='d-box-accent' x='

1.0k views

Understand the problem

Retrieval metrics (recall@k, MRR) plus generation metrics (faithfulness, relevance) — measured separately.

rag-evaluationrecallfaithfulnessevals

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Retrieval eval: recall@k over a labeled set

def recall_at_k(cases, retriever, k=6):
    hits = 0
    for case in cases:                       # {"query": ..., "gold_chunk_ids": [...]}
        got = {c.id for c in retriever(case["query"], k=k)}
        if got & set(case["gold_chunk_ids"]):
            hits += 1
    return hits / len(cases)

# run on every index/chunking/reranker change:
# recall@6  baseline 0.71 → candidate 0.83   <- ship
# recall@6  baseline 0.71 → candidate 0.64   <- do not ship, whatever the demo showed

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

How do you sandbox an agent and enforce least-privilege tool access?

What is semantic caching and when is it safe to use?