Workspace

Cancellation, bounded parallelism and structured concurrency

3 day streakD

Cancellation, bounded parallelism and structured concurrency

Start work so that it can always be stopped and never outlives its owner. Then see how Go and Tokio schedule millions of tasks, and what Amdahl's Law says about speedups.

CurrentLesson 6 of 618 min

When a user closes a chat tab, the model call, the tools it started and their child processes should all stop. In JavaScript, AbortController carries that intent: pass its signal down to fetch, to streams, to your own functions and to run_command, and abort it once.

Structured concurrency

Nathaniel Smith's 2018 essay argued that spawning a task should look like calling a function: the parent waits for every child, and errors and cancellation flow along that tree. Nothing outlives the scope that started it. Python's TaskGroup, Kotlin's coroutine scopes, Java's structured task scopes and Swift's task groups all follow this idea. In JavaScript you build it from AbortSignal plus Promise.all.

bounded-pool.tsTypeScript
/** Run tool calls with at most `limit` in flight. One failure or abort cancels the rest. */
export async function runBounded<T>(
  jobs: ((signal: AbortSignal) => Promise<T>)[],
  limit: number,
  parent: AbortSignal,
): Promise<T[]> {
  const controller = new AbortController();
  const signal = AbortSignal.any([parent, controller.signal]);
  const results = new Array<T>(jobs.length);
  let next = 0;

  async function worker() {
    while (next < jobs.length && !signal.aborted) {
      const index = next++;
      try { results[index] = await jobs[index](signal); }
      catch (error) { controller.abort(error); throw error; }   // cancel siblings
    }
  }

  // The scope waits for every worker; nothing outlives this call.
  await Promise.all(Array.from({ length: Math.min(limit, jobs.length) }, worker));
  signal.throwIfAborted();
  return results;
}

// Usage: per-call timeout plus user cancellation
const signal = AbortSignal.any([userAbort.signal, AbortSignal.timeout(60_000)]);

AbortSignal.any and AbortSignal.timeout are built into Node 20+ and modern browsers.

How other runtimes schedule tasks

RuntimeModelWhat it means for you
Node.jsOne JS thread, event loop, thread pool for some I/ONever block; move CPU work to workers
GoM:N: many goroutines on few OS threads, with work stealingBlocking calls are fine; the runtime parks the goroutine
Rust TokioWork-stealing thread pool running futuresNever block inside async; use spawn_blocking
Python asyncioOne thread, event loop; the GIL limits CPU parallelismUse processes or free-threaded builds for CPU work

Amdahl's Law

If a fraction p of the work can run in parallel on n workers, the best speedup is 1 / ((1 − p) + p / n). With p = 0.9, eight workers give at most 4.7x, and infinite workers give at most 10x. The serial part sets the ceiling. In an agent turn, the model call is the serial part; parallel tools only shrink the rest.

Where you will meet it

Stop buttons

When a user presses stop in a chat UI, the server aborts the provider stream, cancels pending tool calls and kills their process groups.

Fan-out search

Agents that query many sources limit parallelism to stay under rate limits and cancel the stragglers once enough results arrive.

Hedged requests

Send a second request if the first is slow, take whichever returns first, and cancel the other (Module 10).

Check yourself

Answer out loud before you open each one.

What does structured concurrency guarantee?

No task outlives the scope that started it. Errors and cancellation propagate through the tree of tasks.

With 90% parallel work, what is the maximum speedup from 8 workers?

1 / (0.1 + 0.9 / 8) ≈ 4.7x.

Why bound tool-call parallelism?

Rate limits, memory and downstream capacity are finite. An unbounded fan-out turns one request into a spike.

Try it

Use runBounded to fetch 50 URLs with a limit of 5. Abort the parent after 2 seconds and verify, with logs, that no request starts afterwards and every in-flight fetch rejects with an AbortError.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap