mediumAI Engineering

What is LLM-as-judge and what are its known biases?

Q: What is LLM-as-judge and what are its known biases?

**TL;DR.** LLM-as-judge = a (usually strong) model grading outputs against a **rubric** where code cannot — helpfulness, faithfulness, tone. It scales judgment to thousands of cases, but carries known biases: **position** (pairwise order), **verbosity** (longer looks better), **self-preference**, leniency drift. Mitigate with anchored rubrics, randomized order, forced justification — and **validate the judge against human labels** before trusting its numbers. output + input + context <rect class='d-box-accent' x='150

324 views

Understand the problem

Using a model to grade model outputs — scalable, but only after you validate the judge.

llm-as-judgeevalsbiasgrading

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Anchored, justified, single-dimension judge

const rubric = [
  "Score FAITHFULNESS of the answer to the provided context. Levels:",
  "3 = every claim is directly supported by the context",
  "2 = minor unsupported details that do not change the meaning",
  "1 = at least one material claim lacks support or contradicts context",
  "First write a 2-3 sentence justification citing specific claims.",
  'Then output exactly: {"score": 1|2|3}',
].join("\n");

const verdict = await judge.complete({          // different family than the system
  model: JUDGE_MODEL_PINNED,
  temperature: 0,
  prompt: rubric + "\n\nContext:\n" + ctx + "\n\nAnswer:\n" + answer,
});
// weekly: judge vs human labels on the calibration set — alert if agreement < 0.85

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

How do you design good tool definitions for an LLM?

How do you scope retrieval with metadata filtering and multi-tenancy?