hardAI Engineering

How do you evaluate and debug agent trajectories?

Q: How do you evaluate and debug agent trajectories?

**TL;DR.** Grade **outcomes** (task success rate on a scenario suite, with cost/steps as guardrail metrics) *and* **process** (right tool, valid args, error recovery, no loops — judged per step over the full trace). Debugging is **trace-first**: log every model input/output and tool result, replay failures with **recorded tool fixtures**, and diff trajectories across versions. trace store every step, every run <polygon class='d-arrow' points='341,79 346,86

874 views

Understand the problem

Beyond final-answer accuracy: step-level grading, tool-call correctness and trace analysis.

agent-evalstrajectoriestracingdebugging

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Scenario eval with fixture replay

for (const scenario of suite) {
  for (let run = 0; run < 3; run++) {                 // nondeterminism: sample, not once
    const trace = await agent.run(scenario.task, {
      tools: scenario.mode === "replay"
        ? fixtureTools(scenario.recording)             // deterministic replay
        : liveTools,
      budget: { steps: 30, usd: 1 },
    });
    results.push({
      id: scenario.id, run,
      success: await scenario.check(trace),            // programmatic outcome
      steps: trace.steps.length,
      cost: trace.costUsd,
      process: gradeSteps(trace),                      // wrong-tool / bad-args / loop flags
    });
  }
}
report(aggregate(results));   // pass-rate ± variance, cost deltas vs baseline

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

When should you split work across multiple agents instead of one?

Why are BLEU and ROUGE poor metrics for LLM outputs, and what replaced them?