easyAI Engineering

What are evals and why does every LLM feature need them?

Q: What are evals and why does every LLM feature need them?

**TL;DR.** Evals are **automated tests for model behaviour**: realistic inputs + grading criteria + graders (code or LLM judges) producing scores. They exist because LLMs are **nondeterministic** and changes have **non-local effects** — without evals every prompt tweak and model upgrade is vibes-tested. They are the highest-leverage asset in an AI product. dataset real + edge cases <text class='d-text' x='196' y='70'

493 views

Understand the problem

The unit tests of AI engineering: measuring nondeterministic behaviour so you can change things safely.

evalstestingqualityllmops

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Minimal eval harness

const cases: EvalCase[] = loadJsonl("golden/triage.jsonl"); // {input, expected, tags}

const results = await Promise.all(cases.map(async (c) => {
  const out = await system.run(c.input);
  return {
    id: c.id, tags: c.tags,
    exact: out.label === c.expected.label,                    // code grader
    faithful: await judge.score("faithfulness", c.input, out) // rubric judge, 1-5
  };
}));

console.table(sliceBy(results, "tags"));
// accuracy overall 0.91 | slice:refunds 0.95 | slice:multilingual 0.72  <- found it

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

What are tokens and why do they matter for cost and limits?

What are hallucinations and what causes them?