Workspace

SIMD, memory ordering, GC and GPU memory

3 day streakD

SIMD, memory ordering, GC and GPU memory

Where hardware is in late 2026: wide vector instructions, weaker memory models on Arm, garbage collectors that trade pauses for throughput, and GPUs limited by memory bandwidth.

CurrentLesson 6 of 616 min

SIMD: one instruction, many values

SIMD registers hold several values at once: 128 bits on Arm NEON, 256 bits on AVX2 and 512 bits on AVX-512 and some Arm SVE parts. One instruction adds, compares or shuffles every lane together. Compilers auto-vectorise simple loops; libraries such as simdjson hand-write vector code for parsing.

Figure 6.1

Scalar versus eight-lane SIMD

Scalar · 8 instructions
a0+b0a1+b1a2+b2a3+b3a4+b4a5+b5a6+b6a7+b7
SIMD · 1 instruction, 8 lanes
a0+b0a1+b1a2+b2a3+b3a4+b4a5+b5a6+b6a7+b7
A scalar loop adds one pair per instruction. A 256-bit register holds eight 32-bit lanes, so one instruction adds eight pairs. simdjson uses the same idea to classify 64 input bytes at a time as quotes, backslashes or structural characters.

simdjson parses JSON at gigabytes per second by finding every quote and bracket in a block with a few vector instructions and bit tricks, instead of a byte-at-a-time state machine. That changes which side of the latency ladder parsing sits on.

Memory ordering: x86 versus Arm

CPUs and compilers reorder memory operations for speed. x86 uses a fairly strong model: stores become visible in order. Arm uses a weaker one and may make two stores visible to another core in either order. Code that worked by accident on x86 can break on Arm servers such as Graviton or on Apple silicon. The fix is never to rely on luck: use atomics with explicit ordering, or a mutex.

publish.cC
// Writer: fill the buffer, then publish a flag
data = compute();
atomic_store_explicit(&ready, 1, memory_order_release);

// Reader: see the flag, then read the data
if (atomic_load_explicit(&ready, memory_order_acquire))
  use(data);   // guaranteed to see compute()'s result

Release/acquire pairs say “everything before this store is visible after that load”. Plain variables promise nothing across cores.

Garbage collection

Most objects die young. Generational collectors exploit this: a small young generation is collected often and cheaply, and survivors are promoted to an old generation collected rarely. V8, the JVM and .NET all work this way. Allocation-heavy code, such as rebuilding a large message array every turn, keeps the young collector busy and causes pauses that show up as jitter in token streams.

NUMA and GPU memory

Large servers have several CPU sockets, each with its own memory. Reading another socket's memory is slower: this is NUMA. Databases and inference servers pin threads and memory to the same node.

For LLM inference, the binding limit is usually GPU memory. Decoding one token for one sequence must read every model weight from high-bandwidth memory, plus that sequence's KV cache. Token rate is therefore bounded by memory bandwidth divided by bytes read per token, and batch size is bounded by how many KV caches fit. Module 13 builds on this.

ResourceWhat limits itWhat engineers do
CPU coreCache misses, branch mispredictsContiguous data, fewer branches, SIMD
Socket memoryNUMA distancePin threads and memory together
GPUHBM bandwidth and capacityBatching, quantisation, KV cache paging

Where you will meet it

Fast parsers

simdjson, simdutf and Rust's memchr crate use SIMD to scan text. Node's UTF-8 validation uses simdutf.

Arm migrations

Teams moving services to Arm instances run tests under thread sanitizers to catch missing atomics that x86 forgave.

Inference serving

vLLM and SGLang manage KV cache as pages in GPU memory, because GPU memory, not compute, decides how many requests run at once.

Check yourself

Answer out loud before you open each one.

Why can code with a data race work on x86 but fail on Arm?

x86 keeps stores in order; Arm may reorder them. Code that relied on ordering without atomics loses that guarantee.

Why does decode speed depend on memory bandwidth?

Each new token requires reading all weights and the KV cache. Very little arithmetic is done per byte read, so the memory bus is the bottleneck.

What does a generational GC assume?

Most objects die young, so collecting a small young region often is cheap and frees most garbage.

Try it

Run Node with --trace-gc while an agent loop appends 5,000 messages and rebuilds the prompt array each turn. Count scavenges per turn, then switch to appending in place and compare.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap