Workspace

Little's Law and backpressure

3 day streakD

Little's Law and backpressure

One multiplication tells you how much the system holds open. Learn to size pools and queues with it, and what happens when producers outrun consumers.

DepthLesson 5 of 615 min

Little's Law: the average number of items in a system, L, equals the arrival rate, λ, times the average time each item spends inside, W. It holds for any stable system, whatever the distribution. It turns vague capacity questions into arithmetic.

Figure 5.1

Size an agent service

L = λ × W = 20 × 30 = 600 runs open

0limit 800
Utilisation
75%
Memory for open runs
2.3 GB
Headroom
200 runs

The system is stable. Keep headroom for bursts, since λ and W are averages.

Set the arrival rate and run time. L = λ × W gives the runs held open. If that exceeds your concurrency limit, the excess queues and the queue grows every second until something fails. Memory per run turns L into a capacity plan.

Backpressure

When a producer is faster than a consumer, the difference has to go somewhere. Without backpressure, it goes into a buffer that grows until memory runs out. With backpressure, the slow consumer's state propagates upstream and slows the producer down. TCP does this with its receive window; Node streams do it with the return value of write.

backpressure.tsTypeScript
import { once } from "node:events";
import { Readable } from "node:stream";
import { pipeline } from "node:stream/promises";

// Wrong: ignores backpressure. A slow client makes the buffer grow without limit.
for await (const token of modelStream) res.write(token);

// Right: wait for the socket to drain when write() returns false.
for await (const token of modelStream) {
  if (!res.write(token)) await once(res, "drain");
}

// Simplest: let pipeline() handle it, including errors and cleanup.
await pipeline(Readable.from(modelStream), res);

write returning false means the internal buffer passed its highWaterMark. The data is still queued; you are being asked to pause.

Choose a policy for a full queue

PolicyWhat happensGood for
Block the producerThe caller waits for roomInternal pipelines with a natural slow-down
Reject new workReturn 429 or 503 at oncePublic APIs; the client can retry elsewhere
Drop oldestDiscard stale itemsLive telemetry, progress updates
Grow unboundedMemory rises until a crashNothing. This is the default you must replace.

Where you will meet it

Connection pools

If database queries take 20 ms and you serve 2,000 queries per second, L = 40 connections are busy on average. Size the pool above that, with headroom for variance.

Rate limits

At 600 open runs each making one model call every 5 seconds, you send 120 calls per second. Compare that to your provider tier before launch.

Kafka consumers

Consumer lag is the queue in Little's Law. If it grows steadily, arrival exceeds processing and no buffer size will save you.

Check yourself

Answer out loud before you open each one.

At 20 runs per second and 30 seconds per run, how many runs are open?

600, from L = λW = 20 × 30.

What happens when you ignore res.write() returning false?

Data keeps queuing in memory for the slow client until the process runs out of memory.

Why is an unbounded queue never a real policy?

When arrival exceeds service rate, it grows without limit and eventually crashes, just later and with more lost work.

Try it

Stream 10 MB of tokens to a client that reads 1 KB per second, with and without honouring drain. Watch process.memoryUsage().external in each case.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap