easyAI Engineering

When should you use batch APIs instead of real-time LLM calls?

Q: When should you use batch APIs instead of real-time LLM calls?

**TL;DR.** Batch APIs take a **file of requests** and return results **asynchronously** (typically within hours) at **~50% off**, with separate, far higher rate limits. Use them whenever **no user is waiting** — enrichment, backfills, eval runs, offline generation. Architect pipelines queue-first so batch vs realtime is a flag, not a rewrite. realtime user waiting → seconds, full price, tight rate limits batch file in → hours, ~50% price, h

107 views

Understand the problem

The 50% discount most pipelines ignore: async processing for anything without a user waiting.

batch-apicostasyncpipelines

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Stuck? AI Nudge Available

Get a conceptual hint to guide your logic without spoiling the final implementation.

Study the solution

The solution is waiting

Give it an honest attempt first — then compare your thinking with the full walkthrough.

Read the code

Batch job lifecycle

// 1. build JSONL: one request per line, custom_id = idempotency key
const lines = tickets.map((t) => JSON.stringify({
  custom_id: "ticket-" + t.id,
  params: { model, max_tokens: 50,
            messages: [{ role: "user", content: classifyPrompt(t.text) }] },
}));

// 2. submit and record the batch id
const batch = await client.batches.create({ requests: lines });
await jobs.save({ batchId: batch.id, kind: "ticket-classify" });

// 3. on completion webhook/poll: apply successes, re-batch the failures
for (const r of await client.batches.results(batch.id)) {
  if (r.result.type === "succeeded") await applyLabel(r.custom_id, r.result);
  else failures.push(r.custom_id);
}

Join the discussion

Discussion (0)

No responses yet. Be the first to share what you think.

How do you choose the right model for a task?

How do you build resilient LLM calls with rate limits, retries and fallbacks?