Workspace

Build a run_command tool that cannot hang

3 day streakD

Build a run_command tool that cannot hang

Put the module together into the tool every coding agent needs, then see what Linux added in recent years to make it safer: pidfd, Landlock, EEVDF and sched_ext.

CurrentLesson 7 of 720 min

Here are the four ways an agent's shell tool hangs, all from this module. A complete run_command has to handle each one.

  1. 01
    The pipe is full

    The child wrote more than 64 KiB and the harness is not reading. Always drain stdout and stderr while the child runs.

  2. 02
    It waits for input

    The child reads stdin or opens a pager because it detected a terminal. Use pipes, close stdin, and set the noninteractive environment.

  3. 03
    A grandchild holds the pipe

    The main command exited, but a background process inherited stdout, so EOF never arrives. Watch for process exit, then close the streams.

  4. 04
    The timeout misses children

    SIGTERM reached only the child. Start it in a new session and signal the whole process group, then escalate to SIGKILL.

run-command.tsTypeScript
import { spawn } from "node:child_process";
import { NONINTERACTIVE_ENV } from "./noninteractive-env";

type Result = { exit: number | null; signal: string | null; output: string; truncated: boolean; timedOut: boolean };

export function runCommand(command: string, opts: { cwd: string; timeoutMs: number; maxBytes: number; signal?: AbortSignal }): Promise<Result> {
  return new Promise((resolve) => {
    const child = spawn("bash", ["-c", command], {
      cwd: opts.cwd,
      detached: true,                                  // own process group
      stdio: ["ignore", "pipe", "pipe"],               // no stdin to wait on
      env: { PATH: process.env.PATH ?? "", HOME: opts.cwd, ...NONINTERACTIVE_ENV },
    });
    const chunks: Buffer[] = []; let kept = 0, total = 0, timedOut = false;

    const collect = (chunk: Buffer) => {               // drain both pipes, keep a prefix
      total += chunk.length;
      if (kept < opts.maxBytes) { const part = chunk.subarray(0, opts.maxBytes - kept); chunks.push(part); kept += part.length; }
    };
    child.stdout.on("data", collect);
    child.stderr.on("data", collect);

    const killTree = (sig: NodeJS.Signals) => { try { process.kill(-child.pid!, sig); } catch {} };
    const stop = () => { killTree("SIGTERM"); setTimeout(() => killTree("SIGKILL"), 2000).unref(); };
    const timer = setTimeout(() => { timedOut = true; stop(); }, opts.timeoutMs);
    opts.signal?.addEventListener("abort", stop, { once: true });

    child.on("exit", () => {
      clearTimeout(timer);
      killTree("SIGKILL");                             // end background leftovers
      setTimeout(() => { child.stdout.destroy(); child.stderr.destroy(); }, 500).unref();
    });
    child.on("close", (exit, signal) => {              // all pipes closed or destroyed
      resolve({ exit, signal, output: Buffer.concat(chunks).toString("utf8"), truncated: total > kept, timedOut });
    });
  });
}

Whether to kill background leftovers is a policy choice. A harness that supports npm run dev & needs a separate long-running process tool instead.

Decoding Buffer.concat(chunks) can still split the last character at the byte cap. Reuse the UTF-8-safe truncation from Module 1, and tell the model how many bytes were dropped.

What recent kernels add

FeatureSinceWhy a harness cares
pidfdLinux 5.3 (2019)A descriptor for one process. PIDs are reused; signalling a stale PID can hit an unrelated process. pidfd_send_signal cannot.
cgroup.killLinux 5.14 (2021)Kill every process in a cgroup at once, including ones that escaped the process group with their own setsid.
LandlockLinux 5.13 (2021)An unprivileged process can restrict its own and its children's file access, for example to the workspace only.
EEVDFLinux 6.6 (2023)Replaced CFS as the default scheduler. It orders tasks by virtual deadlines, and a task can ask for shorter slices to get lower latency.
sched_extLinux 6.12 (2024)Load scheduling policies as BPF programs, so fleets can tune scheduling without a kernel rebuild.

Where you will meet it

Codex and Claude Code

Both run model-issued commands through a sandboxed runner with timeouts, output caps and process-tree cleanup.

systemd-run

systemd-run --scope -p MemoryMax=2G cmd places a command in its own cgroup with a memory limit, giving whole-tree kill for free.

Bubblewrap

Flatpak's bwrap builds unprivileged namespaces for one command, a light alternative to a container runtime.

Check yourself

Answer out loud before you open each one.

Your agent's shell tool hangs sometimes. Walk through the causes.

A full pipe nobody drains; a child waiting on stdin or a pager; a grandchild holding the pipe open; a timeout that signals only the child.

Why use a pidfd instead of a PID?

PIDs are recycled. A pidfd refers to exactly one process, so a late signal cannot hit a new process with the same number.

How do you kill descendants that called setsid themselves?

Put the command in its own cgroup and write to cgroup.kill, or kill the cgroup with systemd.

Try it

Build the tool above, then attack it with yes, cat, git log with a PTY, sleep 1000 &, and bash -c 'setsid sleep 1000 &'. Watch each with strace -f and note which control stopped it.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap