Workspace

The latency ladder

3 day streakD

The latency ladder

Every operation has a distance. Put them on one scale and you will know where to spend optimisation effort before you profile anything.

DepthLesson 4 of 610 min

The numbers below span nine orders of magnitude. They are rough and vary by hardware, but their ratios are stable enough to reason with. Multiply them by a billion so that 1 ns feels like 1 second, and the distances become something you can feel.

Figure 4.1

From an L1 hit to the first token

  1. L1 cache read1 ns
  2. L2 cache read4 ns
  3. Main memory read100 ns
  4. NVMe SSD read50 µs
  5. Same-region round trip0.5 ms
  6. Cross-continent round trip150 ms
  7. First token from an LLM1 s
CPU and memoryStorageNetworkModel
Bars use a logarithmic scale, so each grid step is 10 times longer. Switch to human time to see the same numbers as if 1 ns lasted 1 second.

An agent harness lives at the bottom of this ladder. A single model turn costs about a second before the first token, and a tool call to a remote API costs tens to hundreds of milliseconds. Shaving microseconds of CPU from a JSON transform rarely matters next to one avoided round trip.

Rules that follow from the ladder

  1. 01
    Count round trips first

    A request that needs three sequential calls to another continent costs half a second before any server does any work. Batch, pipeline or move closer.

  2. 02
    Reuse connections

    A new HTTPS connection costs extra round trips for TCP and TLS. Keep-alive pools turn those into zero for every request after the first (Module 4).

  3. 03
    Then cut bytes

    Bandwidth is the next cost. Compress large payloads and do not send what the reader will not use (Module 8).

  4. 04
    Only then optimise CPU

    Profile first. When CPU does matter, memory access patterns usually matter more than instruction counts.

Where you will meet it

Parallel tool calls

If a model asks for three independent reads, running them concurrently turns three round trips into one. Most agent APIs let one assistant turn contain several tool calls for this reason.

Regional placement

Putting the harness in the same region as the model API and the database removes cross-region hops from every step of a long run.

N+1 queries

Loading 100 rows and then one query per row is 101 round trips. A join or IN query is one.

Check yourself

Answer out loud before you open each one.

Roughly how many main-memory reads fit in one same-region round trip?

About 5,000: 0.5 ms divided by 100 ns.

A step makes 4 sequential calls across a continent at 150 ms each. What is the floor?

600 ms of pure network time, before any server processing.

Why optimise round trips before CPU in an agent harness?

Model and network latency are millions of times larger than most CPU work the harness does per step.

Try it

Time fetch to an API in your region and one on another continent, 20 times each with a reused connection. Then time an in-memory JSON.parse of a typical tool result. Put all three on the ladder.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap