Little's Law and backpressure
One multiplication tells you how much the system holds open. Learn to size pools and queues with it, and what happens when producers outrun consumers.
Little's Law: the average number of items in a system, L, equals the arrival rate, λ, times the average time each item spends inside, W. It holds for any stable system, whatever the distribution. It turns vague capacity questions into arithmetic.
Size an agent service
L = λ × W = 20 × 30 = 600 runs open
- Utilisation
- 75%
- Memory for open runs
- 2.3 GB
- Headroom
- 200 runs
The system is stable. Keep headroom for bursts, since λ and W are averages.
Backpressure
When a producer is faster than a consumer, the difference has to go somewhere. Without backpressure, it goes into a buffer that grows until memory runs out. With backpressure, the slow consumer's state propagates upstream and slows the producer down. TCP does this with its receive window; Node streams do it with the return value of write.
import { once } from "node:events";
import { Readable } from "node:stream";
import { pipeline } from "node:stream/promises";
// Wrong: ignores backpressure. A slow client makes the buffer grow without limit.
for await (const token of modelStream) res.write(token);
// Right: wait for the socket to drain when write() returns false.
for await (const token of modelStream) {
if (!res.write(token)) await once(res, "drain");
}
// Simplest: let pipeline() handle it, including errors and cleanup.
await pipeline(Readable.from(modelStream), res);write returning false means the internal buffer passed its highWaterMark. The data is still queued; you are being asked to pause.
Choose a policy for a full queue
| Policy | What happens | Good for |
|---|---|---|
| Block the producer | The caller waits for room | Internal pipelines with a natural slow-down |
| Reject new work | Return 429 or 503 at once | Public APIs; the client can retry elsewhere |
| Drop oldest | Discard stale items | Live telemetry, progress updates |
| Grow unbounded | Memory rises until a crash | Nothing. This is the default you must replace. |
Where you will meet it
If database queries take 20 ms and you serve 2,000 queries per second, L = 40 connections are busy on average. Size the pool above that, with headroom for variance.
At 600 open runs each making one model call every 5 seconds, you send 120 calls per second. Compare that to your provider tier before launch.
Consumer lag is the queue in Little's Law. If it grows steadily, arrival exceeds processing and no buffer size will save you.
Check yourself
Answer out loud before you open each one.
At 20 runs per second and 30 seconds per run, how many runs are open?
600, from L = λW = 20 × 30.
What happens when you ignore res.write() returning false?
Data keeps queuing in memory for the slow client until the process runs out of memory.
Why is an unbounded queue never a real policy?
When arrival exceeds service rate, it grows without limit and eventually crashes, just later and with more lost work.
Try it
Stream 10 MB of tokens to a client that reads 1 KB per second, with and without honouring drain. Watch process.memoryUsage().external in each case.