SIMD, memory ordering, GC and GPU memory
Where hardware is in late 2026: wide vector instructions, weaker memory models on Arm, garbage collectors that trade pauses for throughput, and GPUs limited by memory bandwidth.
SIMD: one instruction, many values
SIMD registers hold several values at once: 128 bits on Arm NEON, 256 bits on AVX2 and 512 bits on AVX-512 and some Arm SVE parts. One instruction adds, compares or shuffles every lane together. Compilers auto-vectorise simple loops; libraries such as simdjson hand-write vector code for parsing.
Scalar versus eight-lane SIMD
simdjson parses JSON at gigabytes per second by finding every quote and bracket in a block with a few vector instructions and bit tricks, instead of a byte-at-a-time state machine. That changes which side of the latency ladder parsing sits on.
Memory ordering: x86 versus Arm
CPUs and compilers reorder memory operations for speed. x86 uses a fairly strong model: stores become visible in order. Arm uses a weaker one and may make two stores visible to another core in either order. Code that worked by accident on x86 can break on Arm servers such as Graviton or on Apple silicon. The fix is never to rely on luck: use atomics with explicit ordering, or a mutex.
// Writer: fill the buffer, then publish a flag
data = compute();
atomic_store_explicit(&ready, 1, memory_order_release);
// Reader: see the flag, then read the data
if (atomic_load_explicit(&ready, memory_order_acquire))
use(data); // guaranteed to see compute()'s resultRelease/acquire pairs say “everything before this store is visible after that load”. Plain variables promise nothing across cores.
Garbage collection
Most objects die young. Generational collectors exploit this: a small young generation is collected often and cheaply, and survivors are promoted to an old generation collected rarely. V8, the JVM and .NET all work this way. Allocation-heavy code, such as rebuilding a large message array every turn, keeps the young collector busy and causes pauses that show up as jitter in token streams.
NUMA and GPU memory
Large servers have several CPU sockets, each with its own memory. Reading another socket's memory is slower: this is NUMA. Databases and inference servers pin threads and memory to the same node.
For LLM inference, the binding limit is usually GPU memory. Decoding one token for one sequence must read every model weight from high-bandwidth memory, plus that sequence's KV cache. Token rate is therefore bounded by memory bandwidth divided by bytes read per token, and batch size is bounded by how many KV caches fit. Module 13 builds on this.
| Resource | What limits it | What engineers do |
|---|---|---|
| CPU core | Cache misses, branch mispredicts | Contiguous data, fewer branches, SIMD |
| Socket memory | NUMA distance | Pin threads and memory together |
| GPU | HBM bandwidth and capacity | Batching, quantisation, KV cache paging |
Where you will meet it
simdjson, simdutf and Rust's memchr crate use SIMD to scan text. Node's UTF-8 validation uses simdutf.
Teams moving services to Arm instances run tests under thread sanitizers to catch missing atomics that x86 forgave.
vLLM and SGLang manage KV cache as pages in GPU memory, because GPU memory, not compute, decides how many requests run at once.
Check yourself
Answer out loud before you open each one.
Why can code with a data race work on x86 but fail on Arm?
x86 keeps stores in order; Arm may reorder them. Code that relied on ordering without atomics loses that guarantee.
Why does decode speed depend on memory bandwidth?
Each new token requires reading all weights and the KV cache. Very little arithmetic is done per byte read, so the memory bus is the bottleneck.
What does a generational GC assume?
Most objects die young, so collecting a small young region often is cheap and frees most garbage.
Try it
Run Node with --trace-gc while an agent loop appends 5,000 messages and rebuilds the prompt array each turn. Count scavenges per turn, then switch to appending in place and compare.