mediumAI Engineering

How do you estimate and control LLM costs in production?

Q: How do you estimate and control LLM costs in production?

**TL;DR.** Cost = **input tokens × input rate + output tokens × output rate**, summed over calls — and agent loops re-send growing context every turn. Levers in impact order: **route to smaller models**, **prompt caching**, **trim context**, **cap outputs**, **batch APIs**. Meter per feature/user; alert on budgets before the invoice does. route easy tasks to small models prompt caching (~90% off prefix) trim context (RAG > stuffin

831 views

Understand the problem

Token math, the levers that actually move the bill, and per-feature cost accounting.

costtokensoptimizationllmops

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Per-request cost metering

const res = await client.messages.create(reqArgs);

const u = res.usage;   // input_tokens, output_tokens, cache_read_input_tokens...
await metrics.record({
  feature: "support-triage",
  tenant: ctx.tenantId,
  model: reqArgs.model,
  inputTokens: u.input_tokens,
  cachedTokens: u.cache_read_input_tokens ?? 0,
  outputTokens: u.output_tokens,
  usd: price(reqArgs.model, u),
});
// alerts: tenant > daily ceiling, feature > weekly budget, cache-hit-rate < 60%

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

What is the difference between fine-tuning, LoRA and prompt engineering?