hardAI Engineering

How do you monitor LLM quality drift in production?

Q: How do you monitor LLM quality drift in production?

**TL;DR.** Quality degrades **without any deploy**: user inputs drift, providers update models/infrastructure, upstream data changes. Monitor continuously: **LLM-judge scoring on sampled live traffic**, proxy signals (thumbs-down, retry/rephrase, guardrail + repair triggers, refusal rate, output-length shifts), and **alerts on deltas vs a rolling baseline**. Pin versions where offered; re-run the suite on provider updates; fold drifted failures into the golden set. live traffic sample 2-5% <polygon class='d-arrow' points='154,41 162,46 154,51'/

347 views

Understand the problem

Detecting silent degradation: input drift, provider-side changes and feedback-signal dashboards.

driftmonitoringproductionquality

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Continuous sampled scoring job

// hourly cron: score a sample of the last hour's traces
const sample = await traces.sample({ lastHours: 1, rate: 0.03, feature: "support" });

for (const t of sample) {
  const score = await judge.score("faithfulness", t.context, t.output);
  await qualityTs.write({ feature: t.feature, model: t.model, score, at: t.at });
}

// alerting: 7-day rolling mean vs trailing 24h, per feature
// fires when delta > 2σ — catches provider-side shifts with no deploy marker
await alerts.evaluate("quality.faithfulness", { window: "24h", baseline: "7d", sigma: 2 });

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

How do you validate and repair structured LLM output at runtime?

What techniques reduce LLM latency at each stage of a request?