The latency ladder
Every operation has a distance. Put them on one scale and you will know where to spend optimisation effort before you profile anything.
The numbers below span nine orders of magnitude. They are rough and vary by hardware, but their ratios are stable enough to reason with. Multiply them by a billion so that 1 ns feels like 1 second, and the distances become something you can feel.
From an L1 hit to the first token
- L1 cache read1 ns
- L2 cache read4 ns
- Main memory read100 ns
- NVMe SSD read50 µs
- Same-region round trip0.5 ms
- Cross-continent round trip150 ms
- First token from an LLM1 s
An agent harness lives at the bottom of this ladder. A single model turn costs about a second before the first token, and a tool call to a remote API costs tens to hundreds of milliseconds. Shaving microseconds of CPU from a JSON transform rarely matters next to one avoided round trip.
Rules that follow from the ladder
- 01Count round trips first
A request that needs three sequential calls to another continent costs half a second before any server does any work. Batch, pipeline or move closer.
- 02Reuse connections
A new HTTPS connection costs extra round trips for TCP and TLS. Keep-alive pools turn those into zero for every request after the first (Module 4).
- 03Then cut bytes
Bandwidth is the next cost. Compress large payloads and do not send what the reader will not use (Module 8).
- 04Only then optimise CPU
Profile first. When CPU does matter, memory access patterns usually matter more than instruction counts.
Where you will meet it
If a model asks for three independent reads, running them concurrently turns three round trips into one. Most agent APIs let one assistant turn contain several tool calls for this reason.
Putting the harness in the same region as the model API and the database removes cross-region hops from every step of a long run.
Loading 100 rows and then one query per row is 101 round trips. A join or IN query is one.
Check yourself
Answer out loud before you open each one.
Roughly how many main-memory reads fit in one same-region round trip?
About 5,000: 0.5 ms divided by 100 ns.
A step makes 4 sequential calls across a continent at 150 ms each. What is the floor?
600 ms of pure network time, before any server processing.
Why optimise round trips before CPU in an agent harness?
Model and network latency are millions of times larger than most CPU work the harness does per step.
Try it
Time fetch to an API in your region and one on another continent, 20 times each with a reused connection. Then time an in-memory JSON.parse of a typical tool result. Put all three on the ladder.