mediumAI Engineering

How do you catch regressions when changing a prompt or upgrading a model?

Q: How do you catch regressions when changing a prompt or upgrading a model?

**TL;DR.** Run the eval suite on **every change** and compare to baseline — **overall and per-slice**, because aggregate parity can hide a badly regressing slice. Diff individual outputs to see *what* moved. For model upgrades: re-tune prompts, **re-validate judges**, then **canary** with live quality monitoring. No change ships without an eval delta. aggregate: looks fine v41: 0.89 → v42: 0.89 per-slice: it is not refunds 0.92 → 0.95 <rect cla

141 views

Understand the problem

Eval gates, side-by-side diffs and canary rollouts for changes with non-local effects.

regression-testingevalsmodel-upgradeci

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

CI gate with slice thresholds

npm run eval -- --candidate prompts/triage@HEAD --baseline prompts/triage@main \
  --dataset golden/triage.jsonl --runs 3

# eval-report (excerpt)
# overall          0.89 -> 0.90   (+0.01)  OK
# slice:refunds    0.92 -> 0.95   (+0.03)  OK
# slice:multiling  0.84 -> 0.62   (-0.22)  FAIL  max-drop 0.05
# flipped cases: 14 (7 improved, 7 regressed) -> diffs in artifacts/case-diffs.md
exit 1   # merge blocked until the multilingual slice is addressed

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

What are guardrails and how do you implement them around a model?

How does function calling (tool use) work under the hood?