Sign in
Skip to solution
easyAI Engineering

What are evals and why does every LLM feature need them?

493 views
01

Understand the problem

The unit tests of AI engineering: measuring nondeterministic behaviour so you can change things safely.

evalstestingqualityllmops
02

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Nudge consolestandby

Stuck? Beam a request up — the console returns a conceptual nudge that guides your logic without spoiling the implementation.

03

Study the solution

Evals are automated tests for model behaviour: a dataset of realistic inputs, expected outcomes or grading criteria, and graders (code checks or LLM judges) producing scores. They exist because LLM behaviour is nondeterministic and prompt/model changes have non-local effects — without evals every change is vibes-tested

Solution ready — 2 min read

Classified // press E to declassify

04

Read the code

Minimal eval harness
const cases: EvalCase[] = loadJsonl("golden/triage.jsonl"); // {input, expected, tags}

const results = await Promise.all(cases.map(async (c) => {
  const out = await system.run(c.input);
  return {
    id: c.id, tags: c.tags,
    exact: out.label === c.expected.label,                    // code grader
    faithful: await judge.score("faithfulness", c.input, out) // rubric judge, 1-5
  };
}));

console.table(sliceBy(results, "tags"));
// accuracy overall 0.91 | slice:refunds 0.95 | slice:multilingual 0.72  <- found it
05

Join the discussion

Discussion (0)

Sign in to join the discussion.

No responses yet. Be the first to share what you think.

Transmission complete // awaiting log

KEEP THE
STREAK ALIVE.

Dossier 8 of 80 decoded in the AI Engineering track. One more won't hurt.

Back to track