PHASE 02 · MODULE 07
Serialization.
QUIC moves bytes and Protobuf defines what the bytes mean. Choose a format by who reads it, how the schema will evolve and how much of the data you touch.
JSON repeats every field name in every message. Protobuf replaces names with small numbers agreed in a schema. Columnar and zero-copy formats go further and avoid parsing altogether. And for agents there is a twist: the model reads text and you pay per token, so the right format for a model is rarely the right format for a wire.
Start lesson 1Basics
- JSON grammar
- Base64
- JSON Lines
- Big and little endian
- Why JSON numbers lose precision above 2^53
Depth
- Varints and zigzag
- Tag and wire types
- Schema evolution
- Unknown fields
- Why field numbers are never reused
- Canonical serialization
- Streaming parsers
Current
- SIMD JSON parsing
- Protobuf Editions
- Connect RPC
- Arrow Flight
- JSON Schema as the contract for tool calls
Lessons
- 01JSON, Base64 and JSON Lines in depthJSON looks trivial until a number loses digits, a binary blob grows by a third, or a log file is one giant array. Learn the grammar's edges and the text formats around it.
- 02Protobuf by hand: tags, varints and zigzagEncode a Protobuf message byte by byte. Once you can do it on paper, the format's size, speed and compatibility rules all make sense.
- 03Schema evolution: changing a format without breaking anyoneOld readers meet new data and new readers meet old data, for years. Learn the rules that keep both working, and why field numbers are never reused.
- 04Zero-copy and columnar formatsThe fastest parse is no parse. Learn how FlatBuffers and Cap'n Proto read fields in place, and how Arrow and Parquet lay data out by column for analytics.
- 05Serialization for agents: tokens, schemas and cache-stable promptsModels read text and you pay per token, which rewrites the rules: binary saves nothing the model sees, schemas become tool contracts, and byte-identical serialization decides your cache hit rate.
Module lab
- Write a Protobuf encoder and decoder from scratch in about 100 lines, and check it with
protoc --decode_raw. - Benchmark JSON, gzipped JSON and Protobuf in Node on one of your real payloads.
- Measure your harness's prompt cache-hit rate, then make tool serialization deterministic and measure again.
Interview questions
Is Protobuf always faster than JSON?
No. Language runtime, compression and payload shape decide.
How do you rename or remove a field safely?
Keep the number; mark removed numbers reserved.
Why might two identical requests miss the same prompt cache?
Non-deterministic serialization.
Read
- The Protobuf Encoding guide
- Designing Data-Intensive Applications, chapter 4: Encoding and Evolution, Martin Kleppmann
- Parsing Gigabytes of JSON per Second, Langdale and Lemire
- The Apache Arrow columnar format specification