Workspace

Zero-copy and columnar formats

3 day streakD

Zero-copy and columnar formats

The fastest parse is no parse. Learn how FlatBuffers and Cap'n Proto read fields in place, and how Arrow and Parquet lay data out by column for analytics.

DepthLesson 4 of 517 min

JSON and Protobuf both need decoding: walk the bytes and build objects in memory before you can read a field. For a 50 MB message where you need one field, that is a lot of wasted work. Two families of formats avoid it.

Zero-copy: read in place

FlatBuffers and Cap'n Proto lay data out in the buffer the same way it will be used in memory, with offsets instead of pointers. Reading a field is pointer arithmetic on the received bytes. Nothing is copied or allocated, so access time is independent of message size. The price: larger messages, awkward building APIs, and care needed to validate untrusted buffers.

Columnar: one array per field

Row formats store each record's fields together. Columnar formats store each field's values together across all records. A query that sums one column then reads only that column's bytes, contiguously, the cache-friendly pattern from Module 1.

Figure 4.1

The same table, by row and by column

Row layout (JSON, Protobuf, most databases) · 4 separate reads
id1toolgreplatency_ms120oktrueid2toolreadlatency_ms35oktrueid3toolbashlatency_ms4200okfalseid4toolgreplatency_ms95oktrue
Columnar layout (Arrow, Parquet) · 1 separate read
id1id2id3id4toolgreptoolreadtoolbashtoolgreplatency_ms120latency_ms35latency_ms4200latency_ms95oktrueoktrueokfalseoktrue

Columnar reads one contiguous run of latencies and skips every other field. Row layout must step over ids, names and flags to reach each latency.

Four tool-call records stored row by row and column by column. Highlight a query: “average latency” touches every row's latency in row layout, scattered among other fields, but one contiguous run in columnar layout.
FormatWhereCore ideaBest fit
Apache ArrowIn memory, between processesStandard columnar layout; no conversion between toolsAnalytics engines, data frames, Arrow Flight RPC
ParquetOn diskColumnar, encoded and compressed per column, with statistics per chunkData lakes and analytical storage
FlatBuffers / Cap'n ProtoMessagesRead fields in place, no decode stepLarge messages read partially, games, mobile

Why columns compress so well

Values in one column look alike: the same few tool names, timestamps that increase steadily, small integers. Parquet applies dictionary encoding (store each distinct string once, then small indexes), run-length encoding (5,000 identical values become one value and a count), delta encoding (store differences between timestamps) and bit-packing (use 3 bits for values below 8), then a general codec on top (Module 8).

Worked exampleA tool-name column of one million rows
Raw strings, average 12 bytes
≈ 12 MB
Dictionary of 40 distinct names
≈ 480 bytes
Indexes bit-packed at 6 bits each
1,000,000 × 6 ÷ 8 ≈ 750 KB
With runs of the same tool, run-length encoded
Often far smaller still

Over 15x before general compression, because the encoding exploits what the column is.

Go deeperPredicate pushdown: skipping data you never read

Parquet files store minimum and maximum values for each column in each row group. A query for latency_ms > 5000 can skip every row group whose maximum is below 5,000 without decompressing it. Engines such as DuckDB push filters down into the file reader this way.

That makes Parquet in object storage a reasonable database for agent traces: write runs as Parquet files partitioned by day, and query them with DuckDB directly from S3.

Where you will meet it

Trace analytics

Agent observability platforms store spans in columnar engines such as ClickHouse so cost-per-task queries scan only the needed columns.

Data frames

Pandas, Polars and DuckDB exchange Arrow data without copying, so a pipeline can hop between tools cheaply.

Games and mobile

FlatBuffers is popular where parsing time and allocation directly affect frame rate or battery.

Words to know

Zero-copy
Using data in place without copying or decoding it.
Columnar
Storing each field's values together across records.
Dictionary encoding
Replacing repeated values with indexes into a table of distinct values.
Row group
A chunk of rows in a Parquet file with its own per-column statistics.

Check yourself

Answer out loud before you open each one.

What does zero-copy mean for a format?

Fields are read directly from the received bytes via offsets, without decoding into new objects.

Why are analytical scans faster on columnar data?

They read only the needed columns, contiguously, with good cache behaviour and high compression.

Name three encodings Parquet uses before general compression.

Dictionary, run-length, delta and bit-packing (any three).

Try it

Export some JSON Lines traces to Parquet with DuckDB (COPY (SELECT * FROM 'trace.jsonl') TO 'trace.parquet'). Compare file sizes and the time of a SELECT avg(latency_ms) on each.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap