hardAI Engineering

How do you validate and repair structured LLM output at runtime?

Q: How do you validate and repair structured LLM output at runtime?

**TL;DR.** Never trust parsed output: validate against a **schema plus semantic rules** (enums, ranges, cross-field constraints). On failure run a **bounded repair loop** — re-prompt with the specific errors, once or twice — then **fall back deterministically** (constrained decoding, defaults, human queue). A rising repair rate is an early drift alarm. LLM out parsed JSON validate

480 views

Understand the problem

Schema validation, bounded retries with error feedback, and safe fallbacks when repair fails.

validationstructured-outputrepair-loopreliability

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Bounded repair with semantic checks

async function extractInvoice(text: string): Promise<Invoice> {
  let lastErrors = "";
  for (let attempt = 0; attempt < 3; attempt++) {
    const raw = await model.extract(text, lastErrors);       // errors appended on retry
    const parsed = InvoiceSchema.safeParse(raw);
    if (parsed.success) {
      const sem = await semanticChecks(parsed.data);          // vendor exists? totals add up?
      if (sem.ok) { metrics.repairAttempts(attempt); return parsed.data; }
      lastErrors = sem.errors.join("; ");
    } else {
      lastErrors = parsed.error.issues.map((i) => i.path + ": " + i.message).join("; ");
    }
  }
  await reviewQueue.push({ text, lastErrors });               // deterministic fallback
  throw new NeedsHumanReview();
}

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

What is prompt injection and how do you defend against it?

How do you monitor LLM quality drift in production?