Workspace

Windows, bandwidth-delay and small-write stalls

3 day streakD

Windows, bandwidth-delay and small-write stalls

Throughput is window divided by round-trip time. Learn the bandwidth-delay product, why a fat pipe can feel slow, and the classic Nagle plus delayed-ACK stall.

DepthLesson 6 of 720 min

TCP never sends without limit. At any moment, the sender may have at most a window of unacknowledged bytes in flight. Then it must wait for acknowledgements. Two windows apply, and the smaller wins: the receiver's flow control window, which protects the receiver's buffer, and the sender's congestion window, which protects the network.

The bandwidth-delay product

If the window is W bytes and a round trip takes RTT seconds, the sender can deliver at most W per RTT. To keep a link of bandwidth B full, the window must be at least B × RTT. That product is the bandwidth-delay product: the number of bytes that fit “in the pipe” at once.

Figure 6.1

Why a fat pipe can feel slow

Bandwidth-delay product
13 MB
Throughput
5.2 Mbps
Link used
1%

Window-limited: the sender waits on ACKs with the link 99% idle. It needs a 13 MB window.

Set the link speed, round-trip time and window. Throughput is the smaller of the link speed and window ÷ RTT. On long paths a small window wastes most of the link, whatever you paid for.
Worked example1 Gbps across an ocean with a 64 KB window
Bandwidth
1 Gbps = 125,000,000 bytes/s
Round trip
100 ms
Bandwidth-delay product
125,000,000 × 0.1 = 12.5 MB must be in flight
Actual window
64 KB = 65,535 bytes
Throughput
65,535 ÷ 0.1 s = 655 KB/s ≈ 5.2 Mbps

About 0.5% of the link. Window scaling, auto-tuned buffers and parallel connections exist to fix exactly this.

The original TCP header has a 16-bit window field, limiting windows to 64 KB. The window scale option, negotiated in the handshake, multiplies it by up to 2^14. Modern kernels auto-tune receive buffers into the megabytes. A missing or stripped window scale option, or a small fixed SO_RCVBUF, brings the 64 KB ceiling back.

Nagle's algorithm and delayed ACKs

Two old optimisations each make sense alone and stall together. Nagle's algorithm on the sender: if there is unacknowledged small data in flight, hold new small writes until the ACK arrives, then send them together. Delayed ACK on the receiver: wait up to tens of milliseconds before acknowledging, hoping to piggyback the ACK on a reply.

  1. 01
    Client writes a small header

    Sent immediately; nothing is in flight yet.

  2. 02
    Client writes the small body

    Nagle holds it: the header is still unacknowledged.

  3. 03
    Server receives the header

    No full request yet, so no reply to piggyback on. It delays its ACK, often 40 ms on Linux and up to 200 ms elsewhere.

  4. 04
    Delayed-ACK timer fires

    Server ACKs; Nagle releases the body; the request finally completes.

The fix is to stop doing small separate writes: build the whole request and write it once. Or set TCP_NODELAY to disable Nagle on latency-sensitive connections. Node, Go and most HTTP libraries set it by default; custom protocols often forget.

nodelay.tsTypeScript
import net from "node:net";

const socket = net.connect({ host, port, noDelay: true });  // disable Nagle

// Better still: one write per message, not one per field
socket.write(Buffer.concat([header, body]));

noDelay sends small writes immediately. Combining writes still helps: fewer packets, fewer system calls.

Go deeperReceive buffers, zero windows and slow readers

The receiver advertises how much free buffer it has. If the application stops reading, the buffer fills and the advertised window drops to zero. The sender stops and periodically probes. This is TCP's backpressure, and it reaches all the way into your program: a Node socket.write eventually returns false, and the stream asks you to wait for drain (Module 3).

That chain is why a slow client can hold a server's memory. Each stalled connection keeps its unsent bytes in user space if the server ignores backpressure. Servers bound per-connection buffers and drop clients that stay at zero window for too long.

Where you will meet it

Cross-region transfers

Copying data between continents with one TCP stream is often window-limited. Tools such as rclone and cloud SDKs open several parallel streams.

Database drivers

Drivers that write a query's parts separately can hit Nagle and delayed-ACK stalls, showing up as a suspicious 40 ms floor on every query.

Interactive protocols

SSH and terminal sessions set TCP_NODELAY so each keystroke is sent immediately.

Words to know

Flow control window
How much the receiver can buffer; it protects the receiver.
Congestion window
How much the sender allows in flight; it protects the network.
Bandwidth-delay product
Bandwidth × RTT: the bytes needed in flight to keep a link full.
TCP_NODELAY
A socket option that disables Nagle's algorithm.

Check yourself

Answer out loud before you open each one.

Why is a large download slow on a long link?

Throughput is capped at window ÷ RTT. If the window is smaller than the bandwidth-delay product, the link sits idle.

What is the bandwidth-delay product of 100 Mbps at 50 ms RTT?

12,500,000 bytes/s × 0.05 s = 625 KB.

Describe the Nagle and delayed-ACK stall.

The sender holds a small write until the previous one is acknowledged, while the receiver delays the ACK hoping to piggyback it. Both wait until the delayed-ACK timer fires.

Try it

On Linux, add 100 ms of delay with sudo tc qdisc add dev lo root netem delay 50ms and run iperf3 over loopback with -w 64K and then without -w. Remove the delay with tc qdisc del dev lo root.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap