mediumAI Engineering

How do you build a golden dataset for evaluating an LLM feature?

Q: How do you build a golden dataset for evaluating an LLM feature?

**TL;DR.** Source cases from **reality** (sampled traffic + user-flagged failures) plus synthetic coverage of rare-but-important scenarios; **50-200 well-chosen cases beat thousands of random ones**. Label with per-case criteria, adjudicate disagreements, **version it like code**, and keep folding in new production failures. sampled traffic flagged failures synthetic edge cases <line class='d-edge' x1='144' y

1.1k views

Understand the problem

Sourcing, labelling and maintaining the test set your whole quality process hangs on.

golden-datasetevalslabellingquality

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

A golden case with mechanical criteria

// golden/support.jsonl — one JSON per line
{
  "id": "case-0142",
  "source": "prod-2026-06-18/thumbs-down",
  "input": "hey i got charged twice this month?? fix asap",
  "tags": ["billing", "duplicate-charge", "angry-tone"],
  "criteria": {
    "must_route": "billing_team",
    "must_include": ["duplicate charge acknowledged", "refund timeline"],
    "must_not": ["asking user to re-explain", "policy quote without action"],
    "tone": "empathetic, no corporate boilerplate"
  },
  "labelers": ["arv", "nk"], "agreement": "adjudicated-v2"
}

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

How do you get reliable structured JSON output from an LLM?

What is the Model Context Protocol (MCP) and what problem does it solve?