Sign in
Skip to solution
mediumAI Engineering

How do you build a golden dataset for evaluating an LLM feature?

1.1k views
01

Understand the problem

Sourcing, labelling and maintaining the test set your whole quality process hangs on.

golden-datasetevalslabellingquality
02

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Nudge consolestandby

Stuck? Beam a request up — the console returns a conceptual nudge that guides your logic without spoiling the implementation.

03

Study the solution

Source cases from reality: sampled production traffic (especially user-flagged failures and edge cases), plus synthetic cases for coverage of rare-but-important scenarios; 50-200 well-chosen cases beat thousands of random ones. Label with clear per-case criteria (expected answer, must-include facts, rubric) and adjudic

Solution ready — 2 min read

Classified // press E to declassify

04

Read the code

A golden case with mechanical criteria
// golden/support.jsonl — one JSON per line
{
  "id": "case-0142",
  "source": "prod-2026-06-18/thumbs-down",
  "input": "hey i got charged twice this month?? fix asap",
  "tags": ["billing", "duplicate-charge", "angry-tone"],
  "criteria": {
    "must_route": "billing_team",
    "must_include": ["duplicate charge acknowledged", "refund timeline"],
    "must_not": ["asking user to re-explain", "policy quote without action"],
    "tone": "empathetic, no corporate boilerplate"
  },
  "labelers": ["arv", "nk"], "agreement": "adjudicated-v2"
}
05

Join the discussion

Discussion (0)

Sign in to join the discussion.

No responses yet. Be the first to share what you think.

Transmission complete // awaiting log

KEEP THE
STREAK ALIVE.

Dossier 20 of 80 decoded in the AI Engineering track. One more won't hurt.

Back to track