Zero-copy and columnar formats
The fastest parse is no parse. Learn how FlatBuffers and Cap'n Proto read fields in place, and how Arrow and Parquet lay data out by column for analytics.
JSON and Protobuf both need decoding: walk the bytes and build objects in memory before you can read a field. For a 50 MB message where you need one field, that is a lot of wasted work. Two families of formats avoid it.
Zero-copy: read in place
FlatBuffers and Cap'n Proto lay data out in the buffer the same way it will be used in memory, with offsets instead of pointers. Reading a field is pointer arithmetic on the received bytes. Nothing is copied or allocated, so access time is independent of message size. The price: larger messages, awkward building APIs, and care needed to validate untrusted buffers.
Columnar: one array per field
Row formats store each record's fields together. Columnar formats store each field's values together across all records. A query that sums one column then reads only that column's bytes, contiguously, the cache-friendly pattern from Module 1.
The same table, by row and by column
Columnar reads one contiguous run of latencies and skips every other field. Row layout must step over ids, names and flags to reach each latency.
| Format | Where | Core idea | Best fit |
|---|---|---|---|
| Apache Arrow | In memory, between processes | Standard columnar layout; no conversion between tools | Analytics engines, data frames, Arrow Flight RPC |
| Parquet | On disk | Columnar, encoded and compressed per column, with statistics per chunk | Data lakes and analytical storage |
| FlatBuffers / Cap'n Proto | Messages | Read fields in place, no decode step | Large messages read partially, games, mobile |
Why columns compress so well
Values in one column look alike: the same few tool names, timestamps that increase steadily, small integers. Parquet applies dictionary encoding (store each distinct string once, then small indexes), run-length encoding (5,000 identical values become one value and a count), delta encoding (store differences between timestamps) and bit-packing (use 3 bits for values below 8), then a general codec on top (Module 8).
- Raw strings, average 12 bytes
- ≈ 12 MB
- Dictionary of 40 distinct names
- ≈ 480 bytes
- Indexes bit-packed at 6 bits each
- 1,000,000 × 6 ÷ 8 ≈ 750 KB
- With runs of the same tool, run-length encoded
- Often far smaller still
Over 15x before general compression, because the encoding exploits what the column is.
Go deeperPredicate pushdown: skipping data you never read
Parquet files store minimum and maximum values for each column in each row group. A query for latency_ms > 5000 can skip every row group whose maximum is below 5,000 without decompressing it. Engines such as DuckDB push filters down into the file reader this way.
That makes Parquet in object storage a reasonable database for agent traces: write runs as Parquet files partitioned by day, and query them with DuckDB directly from S3.
Where you will meet it
Agent observability platforms store spans in columnar engines such as ClickHouse so cost-per-task queries scan only the needed columns.
Pandas, Polars and DuckDB exchange Arrow data without copying, so a pipeline can hop between tools cheaply.
FlatBuffers is popular where parsing time and allocation directly affect frame rate or battery.
Words to know
- Zero-copy
- Using data in place without copying or decoding it.
- Columnar
- Storing each field's values together across records.
- Dictionary encoding
- Replacing repeated values with indexes into a table of distinct values.
- Row group
- A chunk of rows in a Parquet file with its own per-column statistics.
Check yourself
Answer out loud before you open each one.
What does zero-copy mean for a format?
Fields are read directly from the received bytes via offsets, without decoding into new objects.
Why are analytical scans faster on columnar data?
They read only the needed columns, contiguously, with good cache behaviour and high compression.
Name three encodings Parquet uses before general compression.
Dictionary, run-length, delta and bit-packing (any three).
Try it
Export some JSON Lines traces to Parquet with DuckDB (COPY (SELECT * FROM 'trace.jsonl') TO 'trace.parquet'). Compare file sizes and the time of a SELECT avg(latency_ms) on each.