Cancellation, bounded parallelism and structured concurrency
Start work so that it can always be stopped and never outlives its owner. Then see how Go and Tokio schedule millions of tasks, and what Amdahl's Law says about speedups.
When a user closes a chat tab, the model call, the tools it started and their child processes should all stop. In JavaScript, AbortController carries that intent: pass its signal down to fetch, to streams, to your own functions and to run_command, and abort it once.
Structured concurrency
Nathaniel Smith's 2018 essay argued that spawning a task should look like calling a function: the parent waits for every child, and errors and cancellation flow along that tree. Nothing outlives the scope that started it. Python's TaskGroup, Kotlin's coroutine scopes, Java's structured task scopes and Swift's task groups all follow this idea. In JavaScript you build it from AbortSignal plus Promise.all.
/** Run tool calls with at most `limit` in flight. One failure or abort cancels the rest. */
export async function runBounded<T>(
jobs: ((signal: AbortSignal) => Promise<T>)[],
limit: number,
parent: AbortSignal,
): Promise<T[]> {
const controller = new AbortController();
const signal = AbortSignal.any([parent, controller.signal]);
const results = new Array<T>(jobs.length);
let next = 0;
async function worker() {
while (next < jobs.length && !signal.aborted) {
const index = next++;
try { results[index] = await jobs[index](signal); }
catch (error) { controller.abort(error); throw error; } // cancel siblings
}
}
// The scope waits for every worker; nothing outlives this call.
await Promise.all(Array.from({ length: Math.min(limit, jobs.length) }, worker));
signal.throwIfAborted();
return results;
}
// Usage: per-call timeout plus user cancellation
const signal = AbortSignal.any([userAbort.signal, AbortSignal.timeout(60_000)]);AbortSignal.any and AbortSignal.timeout are built into Node 20+ and modern browsers.
How other runtimes schedule tasks
| Runtime | Model | What it means for you |
|---|---|---|
| Node.js | One JS thread, event loop, thread pool for some I/O | Never block; move CPU work to workers |
| Go | M:N: many goroutines on few OS threads, with work stealing | Blocking calls are fine; the runtime parks the goroutine |
| Rust Tokio | Work-stealing thread pool running futures | Never block inside async; use spawn_blocking |
| Python asyncio | One thread, event loop; the GIL limits CPU parallelism | Use processes or free-threaded builds for CPU work |
Amdahl's Law
If a fraction p of the work can run in parallel on n workers, the best speedup is 1 / ((1 − p) + p / n). With p = 0.9, eight workers give at most 4.7x, and infinite workers give at most 10x. The serial part sets the ceiling. In an agent turn, the model call is the serial part; parallel tools only shrink the rest.
Where you will meet it
When a user presses stop in a chat UI, the server aborts the provider stream, cancels pending tool calls and kills their process groups.
Agents that query many sources limit parallelism to stay under rate limits and cancel the stragglers once enough results arrive.
Send a second request if the first is slow, take whichever returns first, and cancel the other (Module 10).
Check yourself
Answer out loud before you open each one.
What does structured concurrency guarantee?
No task outlives the scope that started it. Errors and cancellation propagate through the tree of tasks.
With 90% parallel work, what is the maximum speedup from 8 workers?
1 / (0.1 + 0.9 / 8) ≈ 4.7x.
Why bound tool-call parallelism?
Rate limits, memory and downstream capacity are finite. An unbounded fan-out turns one request into a spike.
Try it
Use runBounded to fetch 50 URLs with a limit of 5. Abort the parent after 2 seconds and verify, with logs, that no request starts afterwards and every in-flight fetch rejects with an AbortError.