Sign in
Skip to solution
mediumAI Engineering

How does streaming work in LLM APIs and why does it matter for UX?

1.1k views
01

Understand the problem

SSE token streams, time-to-first-token, and rendering partial responses safely.

streamingsselatencyux
02

Attempt it yourself

Sketch your approach before reading the solution — that's what interviews test.

Nudge consolestandby

Stuck? Beam a request up — the console returns a conceptual nudge that guides your logic without spoiling the implementation.

03

Study the solution

LLM APIs stream responses as server-sent events: the client receives token deltas as they are generated instead of waiting for the full completion, cutting perceived latency from seconds to the time-to-first-token. Engineering concerns: render partial markdown/JSON safely (buffer until parseable or use streaming-tolera

Solution ready — 2 min read

Classified // press E to declassify

04

Read the code

Streaming with cancellation propagated
export async function POST(req: Request) {
  const abort = new AbortController();
  req.signal.addEventListener("abort", () => abort.abort());   // user left → stop paying

  const stream = await client.messages.create(
    { model, max_tokens: 1024, stream: true, messages: await req.json() },
    { signal: abort.signal },
  );

  return new Response(new ReadableStream({
    async start(controller) {
      for await (const event of stream) {
        if (event.type === "content_block_delta")
          controller.enqueue(sse(event.delta.text));           // relay, never accumulate
      }
      controller.close();
    },
  }), { headers: { "Content-Type": "text/event-stream", "X-Accel-Buffering": "no" } });
}
05

Join the discussion

Discussion (0)

Sign in to join the discussion.

No responses yet. Be the first to share what you think.

Transmission complete // awaiting log

KEEP THE
STREAK ALIVE.

Dossier 17 of 80 decoded in the AI Engineering track. One more won't hurt.

Back to track