Workspace

Everything that breaks an LLM stream

3 day streakD

Everything that breaks an LLM stream

A worked catalogue of the six ways a token stream fails between the GPU and the screen, with the fix for each and a client that survives them.

DepthLesson 5 of 620 min

Put the module so far together. A token travels from an inference server through gateways, load balancers, CDNs, NATs and the user's Wi-Fi to a parser in a browser. Each hop can break the stream in its own way.

Figure 5.1

Where each failure happens on the path

  1. Inference server1
  2. API gateway
  3. Proxy or CDN1
  4. Load balancer / NAT1
  5. Phone network1
  6. Client parser2
Where
Proxy or CDN
Symptom
Nothing for 20 s, then the whole answer
Fix
Disable buffering for the route; flush per event.
The path from model to screen with the six failure points marked. Select one to see its symptom, its cause and the fix.
  1. 01
    A read ends mid-event

    Symptom: JSON parse errors on some deltas. Cause: treating a network chunk as an event. Fix: buffer until the blank-line delimiter before parsing.

  2. 02
    A read ends mid-character

    Symptom: � in the middle of words, especially emoji and CJK text. Fix: decode with new TextDecoder().decode(chunk, { stream: true }).

  3. 03
    Tool-call arguments arrive as partial JSON

    Symptom: parse errors on input_json_delta events. Fix: accumulate the fragments and parse when the content block stops, or use an incremental JSON parser for live previews.

  4. 04
    A proxy buffers the whole response

    Symptom: nothing for 20 seconds, then the entire answer at once. Fix: disable buffering for the route, for example X-Accel-Buffering: no on nginx, and avoid response compression middlewares that wait for full chunks.

  5. 05
    A quiet period trips an idle timeout

    Symptom: long tool calls or thinking pauses end in a dropped connection. Fix: comment heartbeats every 15 to 30 seconds.

  6. 06
    The phone changes network at token 900

    Symptom: the stream dies and the answer is lost. HTTP gives you nothing here. Fix: persist events by run ID and offset, and let the client resume (next lesson).

accumulate-tool-input.tsTypeScript
// Tool input arrives as string fragments; only the whole thing is valid JSON.
const partial = new Map<number, string>();   // content block index → JSON text so far

for await (const event of parseSSE(response.body!)) {
  const message = JSON.parse(event.data);
  if (message.type === "content_block_delta" && message.delta.type === "input_json_delta") {
    partial.set(message.index, (partial.get(message.index) ?? "") + message.delta.partial_json);
  }
  if (message.type === "content_block_stop" && partial.has(message.index)) {
    const input = JSON.parse(partial.get(message.index)!);   // safe: the block is complete
    partial.delete(message.index);
    queueToolCall(message.index, input);
  }
}

Event names follow a typical model streaming API; check your provider's reference. The pattern is the same everywhere: accumulate, then parse at a known boundary.

A server-side checklist

  • Flush after every event; do not let a framework buffer the body.
  • Set Cache-Control: no-cache and disable proxy buffering for the route.
  • Send heartbeats more often than the shortest idle timeout on the path.
  • Give every event an ID and keep a log so a client can resume.
  • Stop generating when the client disconnects, and record that it did.
Go deeperBatching tokens for the UI

A model can emit 100 tokens per second. Re-rendering a Markdown view per token is wasted work; the screen updates at most 60 times per second, and re-parsing a growing document each time is quadratic (Module 11).

Clients collect tokens into a buffer and flush it to the UI once per animation frame. The network stream stays fine-grained, while rendering stays at a steady frame rate.

Where you will meet it

Platform defaults

Some hosting platforms and CDNs buffer responses unless the content type is text/event-stream or buffering is disabled per route. Test streaming through the real production path, not just locally.

Compression

Gzip middleware can hold output until it has enough to compress. Exclude streaming routes or flush the compressor per event.

Observability

Log time to first token and inter-token gaps at the client, not just the server, to catch buffering in the middle.

Words to know

Proxy buffering
A proxy collecting a response before forwarding it.
Flush
Forcing buffered output onto the network now.
Partial JSON
A fragment of a JSON document delivered as part of a stream.

Check yourself

Answer out loud before you open each one.

Tokens arrive all at once after 20 seconds. What happened?

A proxy, CDN or compression middleware buffered the response. Disable buffering and flush per event.

Why accumulate tool-call input before parsing?

Deltas are fragments of one JSON document; only the complete text is valid JSON.

What should a server do when the client disconnects mid-stream?

Abort the upstream model call and any tools it started, and record the disconnect.

Try it

Put nginx with default settings in front of a local SSE endpoint and watch tokens arrive in a lump. Then add proxy_buffering off; or the X-Accel-Buffering: no header and compare.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap