Sign in
Skip to solution
mediumAI Engineering

What should you log and trace in an LLM application?

42 views
01

Understand the problem

Observability for nondeterministic systems: traces, token metrics, and quality signals.

observabilitytracingloggingllmops
02

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Nudge consolestandby

Stuck? Beam a request up — the console returns a conceptual nudge that guides your logic without spoiling the implementation.

03

Study the solution

Log the full request lifecycle as a trace: prompt version and rendered prompt, model + parameters, retrieved context, every tool call and result, the response, token counts, latency and cost — with PII handling applied. On top of traces, track aggregate metrics (cost per feature, p95 TTFT, error and refusal rates) and

Solution ready — 2 min read

Classified // press E to declassify

04

Read the code

Trace record worth its storage
await traces.write({
  id: reqId, feature: "support-answer", tenant: ctx.tenantId,
  promptVersion: "support-answer@v42",
  model, params: { temperature: 0.2, maxTokens: 800 },
  renderedPrompt: scrubPii(rendered),          // exact bytes, scrubbed
  retrieved: chunks.map((c) => ({ id: c.id, score: c.score })),
  toolCalls: trace.tools,                       // name, args-hash, ok/err, ms
  output: scrubPii(reply.text),
  usage: reply.usage, latencyMs, costUsd,
  feedback: null,                               // thumbs join later → eval candidates
});
05

Join the discussion

Discussion (0)

Sign in to join the discussion.

No responses yet. Be the first to share what you think.

Transmission complete // awaiting log

KEEP THE
STREAK ALIVE.

Dossier 58 of 80 decoded in the AI Engineering track. One more won't hurt.

Back to track