mediumAI Engineering

What should you log and trace in an LLM application?

Q: What should you log and trace in an LLM application?

**TL;DR.** Log each request as a **trace**: prompt version + rendered prompt, model/params, retrieved context, every tool call/result, response, **token counts, latency, cost** — PII-scrubbed. Add aggregate metrics (cost/feature, p95 TTFT, error+refusal rates) and **quality signals** (feedback, sampled judge scores, guardrail triggers). Traces are tomorrow's eval set. trace: request 8f2c span: retrieval (8 chunks, 112ms) <text class='d-sub' x='150' y='90' text-anchor='mid

41 views

Understand the problem

Observability for nondeterministic systems: traces, token metrics, and quality signals.

observabilitytracingloggingllmops

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Trace record worth its storage

await traces.write({
  id: reqId, feature: "support-answer", tenant: ctx.tenantId,
  promptVersion: "support-answer@v42",
  model, params: { temperature: 0.2, maxTokens: 800 },
  renderedPrompt: scrubPii(rendered),          // exact bytes, scrubbed
  retrieved: chunks.map((c) => ({ id: c.id, score: c.score })),
  toolCalls: trace.tools,                       // name, args-hash, ok/err, ms
  output: scrubPii(reply.text),
  usage: reply.usage, latencyMs, costUsd,
  feedback: null,                               // thumbs join later → eval candidates
});

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

What is the ReAct pattern and why did it become the default for agents?

How should an agent handle tool errors, timeouts and retries?