hardAI Engineering

Why are BLEU and ROUGE poor metrics for LLM outputs, and what replaced them?

Q: Why are BLEU and ROUGE poor metrics for LLM outputs, and what replaced them?

**TL;DR.** BLEU/ROUGE score **n-gram overlap with a reference**, assuming one right wording — but LLM outputs are open-ended: a perfect answer can share almost no n-grams with the reference, and overlapping words can still be factually wrong. Modern practice: **embedding similarity** for closeness-to-reference, **task-specific code checks** where objective, and **rubric-anchored LLM-as-judge** for open-ended quality. ref: "Refunds take 5-7 business days" "Expect your money back <text class='d-sub' x='1

765 views

Understand the problem

From n-gram overlap to semantic similarity and rubric-based judging — matching the metric to the task.

metricsbleurougesemantic-similarityevals

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

The failure, concretely

reference = "Refunds take 5-7 business days"

a = "Expect your money back within about a week"   # correct
b = "Refunds take 5-7 business months"             # dangerously wrong

rouge(a, reference)   # ~0.1  -> "bad answer"      (paraphrase punished)
rouge(b, reference)   # ~0.9  -> "great answer"    (one wrong token ignored)

# modern equivalents:
semantic_sim(a, reference)          # high — survives paraphrase
judge("factual_consistency", b)     # fails — catches the wrong unit

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

How do you evaluate and debug agent trajectories?

How do you version, test and safely roll out prompt changes?