Sign in
Skip to solution
mediumAI Engineering

How do you catch regressions when changing a prompt or upgrading a model?

141 views
01

Understand the problem

Eval gates, side-by-side diffs and canary rollouts for changes with non-local effects.

regression-testingevalsmodel-upgradeci
02

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Nudge consolestandby

Stuck? Beam a request up — the console returns a conceptual nudge that guides your logic without spoiling the implementation.

03

Study the solution

Run the eval suite on every change and compare against the baseline run — overall scores plus per-slice breakdowns (task type, language, difficulty), because aggregate parity can hide a slice regressing badly. Diff individual case outputs for the changed cases to understand what moved, not just how much. For model upgr

Solution ready — 2 min read

Classified // press E to declassify

04

Read the code

CI gate with slice thresholds
npm run eval -- --candidate prompts/triage@HEAD --baseline prompts/triage@main \
  --dataset golden/triage.jsonl --runs 3

# eval-report (excerpt)
# overall          0.89 -> 0.90   (+0.01)  OK
# slice:refunds    0.92 -> 0.95   (+0.03)  OK
# slice:multiling  0.84 -> 0.62   (-0.22)  FAIL  max-drop 0.05
# flipped cases: 14 (7 improved, 7 regressed) -> diffs in artifacts/case-diffs.md
exit 1   # merge blocked until the multilingual slice is addressed
05

Join the discussion

Discussion (0)

Sign in to join the discussion.

No responses yet. Be the first to share what you think.

Transmission complete // awaiting log

KEEP THE
STREAK ALIVE.

Dossier 54 of 80 decoded in the AI Engineering track. One more won't hurt.

Back to track