Workspace

Schema evolution: changing a format without breaking anyone

3 day streakD

Schema evolution: changing a format without breaking anyone

Old readers meet new data and new readers meet old data, for years. Learn the rules that keep both working, and why field numbers are never reused.

DepthLesson 3 of 516 min

Data outlives code. A message written today may be read by a service deployed last month and another deployed next year. During a rolling deploy, old and new versions run side by side and talk to each other. Schema evolution is the set of changes you can make without breaking any of these readers.

CompatibilityMeansNeeded when
BackwardNew code can read data written by old codeReading old stored data after an upgrade
ForwardOld code can read data written by new codeRolling deploys, old clients, rollbacks
FullBothLong-lived storage and public APIs
Figure 3.1

Two versions, four conversations

// v1
int64  id       = 1;
string name     = 2;
sint32 priority = 3;
repeated int32 ranks = 4;
// v2
int64  id       = 1;
string name     = 2;
reserved 3;
repeated int32 ranks = 4;
string idempotency_key = 5;
  • v1 writer → v1 reader

    Same schema. Works.

  • v1 writer → v2 reader

    Field 5 (idempotency_key) is missing, so the reader sees its default, an empty string. Code must handle that.

  • v2 writer → v1 reader

    Field 5 is unknown. The reader skips it using its wire type and keeps the bytes. Works.

  • v2 writer → v2 reader

    Same schema. Works.

  • Old v1 writer → v3 reader that reused number 3

    Field 3 meant priority (sint32); v3 reads it as timeout_ms (int32). Silent wrong data.

Version 2 of a message adds a field and removes another. Each cell shows what happens when a reader of one version meets data from the other: unknown fields are skipped or preserved, missing fields take defaults, and a reused number is the one change that corrupts data.

Rules that keep Protobuf compatible

  1. 01
    Add fields freely, with new numbers

    Old readers skip the unknown tag using its wire type. New readers see the default value in old data.

  2. 02
    Never change a field's number

    The number is the field's identity on the wire. The name is only for code.

  3. 03
    Never reuse a removed number

    Mark it reserved. Old data or old services may still send the old meaning under that number.

  4. 04
    Change types only within wire-compatible groups

    int32, int64 and bool share VARINT, with truncation risks; string and bytes share LEN. Most other changes break.

  5. 05
    Treat rename as free, and removal as expensive

    Renaming affects code and JSON mapping, not the wire. Removal requires reserving.

tool_call_v2.protoText
message ToolCall {
  reserved 3;                 // was: sint32 priority
  reserved "priority";        // also stop the name being reused in JSON

  int64  id      = 1;
  string name    = 2;
  repeated int32 ranks = 4;
  string idempotency_key = 5; // new in v2: old readers skip it
}
Worked exampleWhat goes wrong when a number is reused
v1
field 3 = sint32 priority
v2 removes priority, then a later change adds
field 3 = int32 timeout_ms
An old client sends priority −1
zigzag-encoded as 1
A new server reads field 3 as int32
timeout_ms = 1

No error, just wrong data: a one-millisecond timeout. reserved makes the compiler refuse this change.

Unknown fields

When a reader meets a field number it does not know, Protobuf skips it but also keeps its bytes in the parsed object. If that object is re-serialised, the unknown field goes back out. A proxy built on an old schema can therefore pass new fields through without understanding them.

Go deeperJSON's evolution rules, and Protobuf Editions

JSON has no schema on the wire, so the same principles apply by convention: readers ignore unknown fields, writers only add optional fields, and nobody changes a field's meaning under the same name. A strict JSON Schema validator with additionalProperties: false breaks forward compatibility, which is sometimes the right choice for tool inputs and wrong for API responses.

Protobuf Editions replaces the proto2/proto3 split with per-file editions and per-feature settings, such as whether fields track presence. It lets the format evolve its defaults without another syntax fork.

Where you will meet it

Rolling deploys

Every rolling deploy is a forward and backward compatibility test between adjacent versions.

Schema registries

Kafka users register Avro or Protobuf schemas and let the registry reject incompatible changes before they ship.

Linters

Tools such as buf breaking compare a schema against the previous version in CI and fail on reused numbers or incompatible type changes.

Words to know

Backward compatible
New readers handle old data.
Forward compatible
Old readers handle new data.
reserved
A Protobuf declaration that forbids reusing field numbers or names.
Unknown field
A field a reader's schema does not define; skipped and often preserved.

Check yourself

Answer out loud before you open each one.

How do you rename or remove a field safely?

Rename freely; it does not affect the wire. To remove, keep the number unused forever with reserved.

What does an old reader do with a new field?

Skips it using the wire type, and in Protobuf preserves its bytes for re-serialisation.

Why is reusing a field number dangerous?

Old writers still send the old meaning under that number, and new readers silently misinterpret it.

Try it

Write v1 and v2 of a message, serialise data with each, and read it with the other using protoc. Then reuse a reserved number and see the compiler error.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap