Everything that breaks an LLM stream
A worked catalogue of the six ways a token stream fails between the GPU and the screen, with the fix for each and a client that survives them.
Put the module so far together. A token travels from an inference server through gateways, load balancers, CDNs, NATs and the user's Wi-Fi to a parser in a browser. Each hop can break the stream in its own way.
Where each failure happens on the path
- Inference server1
- API gateway
- Proxy or CDN1
- Load balancer / NAT1
- Phone network1
- Client parser2
- Where
- Proxy or CDN
- Symptom
- Nothing for 20 s, then the whole answer
- Fix
- Disable buffering for the route; flush per event.
- 01A read ends mid-event
Symptom: JSON parse errors on some deltas. Cause: treating a network chunk as an event. Fix: buffer until the blank-line delimiter before parsing.
- 02A read ends mid-character
Symptom:
�in the middle of words, especially emoji and CJK text. Fix: decode withnew TextDecoder().decode(chunk, { stream: true }). - 03Tool-call arguments arrive as partial JSON
Symptom: parse errors on
input_json_deltaevents. Fix: accumulate the fragments and parse when the content block stops, or use an incremental JSON parser for live previews. - 04A proxy buffers the whole response
Symptom: nothing for 20 seconds, then the entire answer at once. Fix: disable buffering for the route, for example
X-Accel-Buffering: noon nginx, and avoid response compression middlewares that wait for full chunks. - 05A quiet period trips an idle timeout
Symptom: long tool calls or thinking pauses end in a dropped connection. Fix: comment heartbeats every 15 to 30 seconds.
- 06The phone changes network at token 900
Symptom: the stream dies and the answer is lost. HTTP gives you nothing here. Fix: persist events by run ID and offset, and let the client resume (next lesson).
// Tool input arrives as string fragments; only the whole thing is valid JSON.
const partial = new Map<number, string>(); // content block index → JSON text so far
for await (const event of parseSSE(response.body!)) {
const message = JSON.parse(event.data);
if (message.type === "content_block_delta" && message.delta.type === "input_json_delta") {
partial.set(message.index, (partial.get(message.index) ?? "") + message.delta.partial_json);
}
if (message.type === "content_block_stop" && partial.has(message.index)) {
const input = JSON.parse(partial.get(message.index)!); // safe: the block is complete
partial.delete(message.index);
queueToolCall(message.index, input);
}
}Event names follow a typical model streaming API; check your provider's reference. The pattern is the same everywhere: accumulate, then parse at a known boundary.
A server-side checklist
- Flush after every event; do not let a framework buffer the body.
- Set
Cache-Control: no-cacheand disable proxy buffering for the route. - Send heartbeats more often than the shortest idle timeout on the path.
- Give every event an ID and keep a log so a client can resume.
- Stop generating when the client disconnects, and record that it did.
Go deeperBatching tokens for the UI
A model can emit 100 tokens per second. Re-rendering a Markdown view per token is wasted work; the screen updates at most 60 times per second, and re-parsing a growing document each time is quadratic (Module 11).
Clients collect tokens into a buffer and flush it to the UI once per animation frame. The network stream stays fine-grained, while rendering stays at a steady frame rate.
Where you will meet it
Some hosting platforms and CDNs buffer responses unless the content type is text/event-stream or buffering is disabled per route. Test streaming through the real production path, not just locally.
Gzip middleware can hold output until it has enough to compress. Exclude streaming routes or flush the compressor per event.
Log time to first token and inter-token gaps at the client, not just the server, to catch buffering in the middle.
Words to know
- Proxy buffering
- A proxy collecting a response before forwarding it.
- Flush
- Forcing buffered output onto the network now.
- Partial JSON
- A fragment of a JSON document delivered as part of a stream.
Check yourself
Answer out loud before you open each one.
Tokens arrive all at once after 20 seconds. What happened?
A proxy, CDN or compression middleware buffered the response. Disable buffering and flush per event.
Why accumulate tool-call input before parsing?
Deltas are fragments of one JSON document; only the complete text is valid JSON.
What should a server do when the client disconnects mid-stream?
Abort the upstream model call and any tools it started, and record the disconnect.
Try it
Put nginx with default settings in front of a local SSE endpoint and watch tokens arrive in a lump. Then add proxy_buffering off; or the X-Accel-Buffering: no header and compare.