Protobuf by hand: tags, varints and zigzag
Encode a Protobuf message byte by byte. Once you can do it on paper, the format's size, speed and compatibility rules all make sense.
A Protobuf schema assigns each field a number. On the wire, a message is a sequence of fields, each written as a tag followed by a value. Names never appear. The tag packs the field number and a wire type that tells the reader how to find the value's end, even for fields it does not know.
syntax = "proto3";
message ToolCall {
int64 id = 1;
string name = 2;
sint32 priority = 3;
repeated int32 ranks = 4;
}| Wire type | Name | Used for | How the reader finds the end |
|---|---|---|---|
| 0 | VARINT | int32, int64, uint, bool, enum, sint | Bytes until one with the top bit clear |
| 1 | I64 | fixed64, double | Exactly 8 bytes |
| 2 | LEN | string, bytes, nested messages, packed repeated | A varint length, then that many bytes |
| 5 | I32 | fixed32, float | Exactly 4 bytes |
Varints
A varint writes an integer 7 bits at a time, least significant group first. Each byte's top bit is a continuation bit: 1 means another byte follows, 0 means this is the last. Small numbers take one byte; the largest 64-bit numbers take ten.
- Tag
- (field 1 << 3) | wire type 0 = 8 =
0x08 - 150 in binary
- 1001 0110
- Split into 7-bit groups, low first
0010110then0000001- Add continuation bits
10010110 =0x96, then00000001 =0x01- Message bytes
08 96 01, 3 bytes- Same data as JSON
{"id":150}, 10 bytes
Names cost nothing on the wire because the schema holds them. That is most of Protobuf's size advantage.
Encode a field yourself
080000100096100101100100000001- Protobuf bytes
- 08 96 01
- Protobuf size
- 3 B
- JSON
{"id":150} - 10 B
Negative numbers and zigzag
Negative int32 values are sign-extended to 64 bits before varint encoding, so −1 becomes ten bytes. Types sint32 and sint64 use zigzag encoding first, mapping small negatives to small positives: 0→0, −1→1, 1→2, −2→3. Then −1 costs one byte. Choose sint for fields that are often negative.
export function encodeVarint(value: bigint): number[] {
const out: number[] = [];
let v = BigInt.asUintN(64, value); // negative int32/int64 → two's complement
do {
let byte = Number(v & 0x7fn);
v >>= 7n;
if (v) byte |= 0x80; // continuation bit
out.push(byte);
} while (v);
return out;
}
export const zigzag32 = (n: number) => ((n << 1) ^ (n >> 31)) >>> 0;
export const tag = (field: number, wireType: number) => encodeVarint(BigInt((field << 3) | wireType));
// field 1 = 150 → [0x08, 0x96, 0x01]
console.log([...tag(1, 0), ...encodeVarint(150n)].map((b) => b.toString(16)));Check your encoder against protoc --decode_raw, which prints field numbers and values from raw bytes without a schema.
Is Protobuf always faster than JSON?
No. Three things narrow the gap. Compression: gzip removes JSON's repeated names well. Runtime: JavaScript engines parse JSON in highly optimised native code, while a Protobuf library may run in JavaScript. Payload shape: large strings cost the same in both. Benchmark your real payload in your real language before switching.
Go deeperField numbers 1 to 15 and packed repeated fields
The tag is itself a varint. With 3 bits for the wire type, field numbers 1 to 15 fit in one tag byte; 16 to 2,047 need two. Put frequently set fields in 1 to 15.
A repeated int32 field could repeat its tag for every element. In proto3, numeric repeated fields are packed by default: one tag with wire type LEN, a total byte length, then the varints back to back. A list of 1,000 small numbers drops from about 2,000 bytes to about 1,003.
Where you will meet it
Every gRPC message is Protobuf behind a 5-byte frame prefix (Module 6).
Systems store Protobuf in databases and logs for compact, schema-checked records that old and new code can both read.
protoc --decode_raw < message.bin reveals the fields of an unknown message, which is often the fastest way to debug a wire problem.
Words to know
- Tag
- Field number and wire type packed into one varint.
- Varint
- A variable-length integer, 7 data bits per byte plus a continuation bit.
- Wire type
- How a field's value is laid out, so readers can skip unknown fields.
- Zigzag
- A mapping that turns small negative integers into small positive ones.
Check yourself
Answer out loud before you open each one.
Encode field 2, the string "hi".
Tag (2 << 3) | 2 = 0x12, length 0x02, then 0x68 0x69: 12 02 68 69.
Why does −1 as int32 take 10 bytes?
It is sign-extended to 64 bits, so the varint must carry all 64 bits, 7 at a time.
Is Protobuf always faster than JSON?
No. Language runtime, compression and payload shape decide; benchmark the real case.
Try it
Encode ToolCall { id: 150, name: "grep" } by hand on paper. Then write the bytes with printf '\x08\x96\x01\x12\x04grep' | protoc --decode_raw.