Schema evolution: changing a format without breaking anyone
Old readers meet new data and new readers meet old data, for years. Learn the rules that keep both working, and why field numbers are never reused.
Data outlives code. A message written today may be read by a service deployed last month and another deployed next year. During a rolling deploy, old and new versions run side by side and talk to each other. Schema evolution is the set of changes you can make without breaking any of these readers.
| Compatibility | Means | Needed when |
|---|---|---|
| Backward | New code can read data written by old code | Reading old stored data after an upgrade |
| Forward | Old code can read data written by new code | Rolling deploys, old clients, rollbacks |
| Full | Both | Long-lived storage and public APIs |
Two versions, four conversations
// v1 int64 id = 1; string name = 2; sint32 priority = 3; repeated int32 ranks = 4;
// v2 int64 id = 1; string name = 2; reserved 3; repeated int32 ranks = 4; string idempotency_key = 5;
- v1 writer → v1 reader
Same schema. Works.
- v1 writer → v2 reader
Field 5 (idempotency_key) is missing, so the reader sees its default, an empty string. Code must handle that.
- v2 writer → v1 reader
Field 5 is unknown. The reader skips it using its wire type and keeps the bytes. Works.
- v2 writer → v2 reader
Same schema. Works.
- Old v1 writer → v3 reader that reused number 3
Field 3 meant priority (sint32); v3 reads it as timeout_ms (int32). Silent wrong data.
Rules that keep Protobuf compatible
- 01Add fields freely, with new numbers
Old readers skip the unknown tag using its wire type. New readers see the default value in old data.
- 02Never change a field's number
The number is the field's identity on the wire. The name is only for code.
- 03Never reuse a removed number
Mark it
reserved. Old data or old services may still send the old meaning under that number. - 04Change types only within wire-compatible groups
int32, int64 and bool share VARINT, with truncation risks; string and bytes share LEN. Most other changes break.
- 05Treat rename as free, and removal as expensive
Renaming affects code and JSON mapping, not the wire. Removal requires reserving.
message ToolCall {
reserved 3; // was: sint32 priority
reserved "priority"; // also stop the name being reused in JSON
int64 id = 1;
string name = 2;
repeated int32 ranks = 4;
string idempotency_key = 5; // new in v2: old readers skip it
}- v1
- field 3 =
sint32 priority - v2 removes priority, then a later change adds
- field 3 =
int32 timeout_ms - An old client sends priority −1
- zigzag-encoded as 1
- A new server reads field 3 as int32
- timeout_ms = 1
No error, just wrong data: a one-millisecond timeout. reserved makes the compiler refuse this change.
Unknown fields
When a reader meets a field number it does not know, Protobuf skips it but also keeps its bytes in the parsed object. If that object is re-serialised, the unknown field goes back out. A proxy built on an old schema can therefore pass new fields through without understanding them.
Go deeperJSON's evolution rules, and Protobuf Editions
JSON has no schema on the wire, so the same principles apply by convention: readers ignore unknown fields, writers only add optional fields, and nobody changes a field's meaning under the same name. A strict JSON Schema validator with additionalProperties: false breaks forward compatibility, which is sometimes the right choice for tool inputs and wrong for API responses.
Protobuf Editions replaces the proto2/proto3 split with per-file editions and per-feature settings, such as whether fields track presence. It lets the format evolve its defaults without another syntax fork.
Where you will meet it
Every rolling deploy is a forward and backward compatibility test between adjacent versions.
Kafka users register Avro or Protobuf schemas and let the registry reject incompatible changes before they ship.
Tools such as buf breaking compare a schema against the previous version in CI and fail on reused numbers or incompatible type changes.
Words to know
- Backward compatible
- New readers handle old data.
- Forward compatible
- Old readers handle new data.
- reserved
- A Protobuf declaration that forbids reusing field numbers or names.
- Unknown field
- A field a reader's schema does not define; skipped and often preserved.
Check yourself
Answer out loud before you open each one.
How do you rename or remove a field safely?
Rename freely; it does not affect the wire. To remove, keep the number unused forever with reserved.
What does an old reader do with a new field?
Skips it using the wire type, and in Protobuf preserves its bytes for re-serialisation.
Why is reusing a field number dangerous?
Old writers still send the old meaning under that number, and new readers silently misinterpret it.
Try it
Write v1 and v2 of a message, serialise data with each, and read it with the other using protoc. Then reuse a reserved number and see the compiler error.