Workspace

Pipes, buffers and EOF

3 day streakD

Pipes, buffers and EOF

A pipe is a small kernel buffer with a reader and a writer. When nobody reads, the writer blocks. When anybody still holds the write end, the reader never sees EOF.

BasicsLesson 3 of 715 min

pipe() returns two descriptors: a read end and a write end. Bytes written to one come out of the other, in order. Between them sits a kernel buffer, 64 KiB by default on Linux. When it fills, write blocks until a reader makes room. When it is empty, read blocks until a writer adds bytes or every write end is closed, which the reader sees as EOF, a read that returns 0.

Figure 3.1

Fill a pipe until the writer blocks

Child · npm testRunning, writing output0 of 160 KiB written
Pipe buffer · 0 / 64 KiB
HarnessReading stdout while it waits0 KiB drained

Each step the child writes 8 KiB and the harness reads up to 12 KiB.

The child has 160 KiB of output. With the harness reading, the buffer stays nearly empty. Stop reading and the buffer fills at 64 KiB; the child then blocks inside write and never exits, while a harness that is waiting for exit waits forever.

Hang 1: the full pipe

A classic deadlock: the harness waits for the child to exit and only then reads its output. The child produces more than 64 KiB, blocks in write, and never exits. Neither side moves. The fix is to read stdout and stderr continuously while the child runs, even if you only keep the first part.

Hang 2: EOF never arrives

EOF arrives only when every descriptor for the write end is closed. Suppose the model runs npm run dev & and the main command exits. The background server inherited stdout, so it still holds the write end. The harness reads forever, waiting for an EOF that comes only when the server dies. Listen for process exit as well as stream end, and decide what to do with descendants that outlive their parent.

collect.tsTypeScript
const MAX = 256 * 1024;                     // keep at most 256 KiB
let kept: Buffer[] = [], keptBytes = 0, totalBytes = 0;

child.stdout.on("data", (chunk: Buffer) => {
  totalBytes += chunk.length;              // always drain the pipe
  if (keptBytes < MAX) {
    const slice = chunk.subarray(0, MAX - keptBytes);
    kept.push(slice); keptBytes += slice.length;
  }
});

// 'exit' fires when the process ends; 'close' waits for all stdio to end.
// A background grandchild can delay 'close' forever.
child.on("exit", () => setTimeout(() => child.stdout.destroy(), 500));

Draining and keeping are separate decisions. Always drain; keep only what you will use.

Where you will meet it

Python's `subprocess`

The documentation warns that wait() with stdout=PIPE can deadlock on large output, and recommends communicate(), which reads both pipes while waiting.

Shell pipelines

In producer | head -1, head exits after one line. The producer's next write fails with SIGPIPE, which is how pipelines stop early.

Agent shell tools

Production tools stream output to the model in chunks and cap what they keep, while draining everything so the child can finish.

Check yourself

Answer out loud before you open each one.

Why does a child hang when the harness reads only after exit?

Its output fills the 64 KiB pipe buffer, write blocks, and the child cannot exit.

The main command exited but the read never ends. Why?

A background descendant inherited the write end of the pipe and still holds it open, so EOF never arrives.

What happens when a process writes to a pipe whose readers have all closed?

It receives SIGPIPE, which terminates it by default, or write fails with EPIPE if the signal is ignored.

Try it

Write a tiny harness that spawns yes | head -c 200000 and calls waitpid before reading. Watch it hang with strace -p. Then fix it by reading while waiting.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap