mediumAI Engineering

When do you use code-based graders versus model-based graders?

Q: When do you use code-based graders versus model-based graders?

**TL;DR.** **Code graders** for anything objectively checkable — schema, exact/contains, numeric tolerance, tests passing, latency budgets: free, deterministic, unarguable. **Model graders** for what code cannot verify — relevance, faithfulness, tone. **Layer them**: code gates first, judge scores the survivors; and keep converting judged criteria into code checks. outputs code gates <text class='d-sub' x='202' y='78' text-anch

969 views

Understand the problem

Deterministic checks vs judged qualities — and why good eval suites layer both.

gradersevalscode-gradersllm-as-judge

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Layered grading, reported separately

async function grade(c: EvalCase, out: Output) {
  const gates = {
    schema: Schema.safeParse(out.json).success,
    citations: citationsResolve(out.text, c.contextIds),
    banned: !BANNED.some((w) => out.text.includes(w)),
    latency: out.ms < 3000,
  };
  if (Object.values(gates).some((v) => !v)) return { gates, judged: null }; // fail fast

  const judged = {
    faithfulness: await judge.score("faithfulness", c, out),   // survivors only
    tone: await judge.score("tone", c, out),
  };
  return { gates, judged };
}

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

What is the Model Context Protocol (MCP) and what problem does it solve?

How do multimodal models handle images, audio and other non-text inputs?