Sign in
Skip to solution
mediumAI Engineering

What is LLM-as-judge and what are its known biases?

324 views
01

Understand the problem

Using a model to grade model outputs — scalable, but only after you validate the judge.

llm-as-judgeevalsbiasgrading
02

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Nudge consolestandby

Stuck? Beam a request up — the console returns a conceptual nudge that guides your logic without spoiling the implementation.

03

Study the solution

LLM-as-judge uses a (usually strong) model with a grading rubric to score outputs where code cannot — helpfulness, faithfulness, tone. It scales human-like judgment to thousands of cases, but has documented biases: position bias in pairwise comparisons (favouring the first/last answer), verbosity bias (longer looks bet

Solution ready — 2 min read

Classified // press E to declassify

04

Read the code

Anchored, justified, single-dimension judge
const rubric = [
  "Score FAITHFULNESS of the answer to the provided context. Levels:",
  "3 = every claim is directly supported by the context",
  "2 = minor unsupported details that do not change the meaning",
  "1 = at least one material claim lacks support or contradicts context",
  "First write a 2-3 sentence justification citing specific claims.",
  'Then output exactly: {"score": 1|2|3}',
].join("\n");

const verdict = await judge.complete({          // different family than the system
  model: JUDGE_MODEL_PINNED,
  temperature: 0,
  prompt: rubric + "\n\nContext:\n" + ctx + "\n\nAnswer:\n" + answer,
});
// weekly: judge vs human labels on the calibration set — alert if agreement < 0.85
05

Join the discussion

Discussion (0)

Sign in to join the discussion.

No responses yet. Be the first to share what you think.

Transmission complete // awaiting log

KEEP THE
STREAK ALIVE.

Dossier 46 of 80 decoded in the AI Engineering track. One more won't hurt.

Back to track