Workspace

Serialization

3 day streakD
Systems syllabus
PHASE 02 · MODULE 07

Serialization.

QUIC moves bytes and Protobuf defines what the bytes mean. Choose a format by who reads it, how the schema will evolve and how much of the data you touch.

JSON repeats every field name in every message. Protobuf replaces names with small numbers agreed in a schema. Columnar and zero-copy formats go further and avoid parsing altogether. And for agents there is a twist: the model reads text and you pay per token, so the right format for a model is rarely the right format for a wire.

5 lessons93 minutes of reading5 illustrations
Start lesson 1
Basics
  • JSON grammar
  • Base64
  • JSON Lines
  • Big and little endian
  • Why JSON numbers lose precision above 2^53
Depth
  • Varints and zigzag
  • Tag and wire types
  • Schema evolution
  • Unknown fields
  • Why field numbers are never reused
  • Canonical serialization
  • Streaming parsers
Current
  • SIMD JSON parsing
  • Protobuf Editions
  • Connect RPC
  • Arrow Flight
  • JSON Schema as the contract for tool calls

Lessons

  1. 01JSON, Base64 and JSON Lines in depthJSON looks trivial until a number loses digits, a binary blob grows by a third, or a log file is one giant array. Learn the grammar's edges and the text formats around it.Basics18 min
  2. 02Protobuf by hand: tags, varints and zigzagEncode a Protobuf message byte by byte. Once you can do it on paper, the format's size, speed and compatibility rules all make sense.Depth22 min
  3. 03Schema evolution: changing a format without breaking anyoneOld readers meet new data and new readers meet old data, for years. Learn the rules that keep both working, and why field numbers are never reused.Depth16 min
  4. 04Zero-copy and columnar formatsThe fastest parse is no parse. Learn how FlatBuffers and Cap'n Proto read fields in place, and how Arrow and Parquet lay data out by column for analytics.Depth17 min
  5. 05Serialization for agents: tokens, schemas and cache-stable promptsModels read text and you pay per token, which rewrites the rules: binary saves nothing the model sees, schemas become tool contracts, and byte-identical serialization decides your cache hit rate.Current20 min

Module lab

  1. Write a Protobuf encoder and decoder from scratch in about 100 lines, and check it with protoc --decode_raw.
  2. Benchmark JSON, gzipped JSON and Protobuf in Node on one of your real payloads.
  3. Measure your harness's prompt cache-hit rate, then make tool serialization deterministic and measure again.

Interview questions

Is Protobuf always faster than JSON?

No. Language runtime, compression and payload shape decide.

How do you rename or remove a field safely?

Keep the number; mark removed numbers reserved.

Why might two identical requests miss the same prompt cache?

Non-deterministic serialization.

Read

  • The Protobuf Encoding guide
  • Designing Data-Intensive Applications, chapter 4: Encoding and Evolution, Martin Kleppmann
  • Parsing Gigabytes of JSON per Second, Langdale and Lemire
  • The Apache Arrow columnar format specification

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap