Workspace

Protobuf by hand: tags, varints and zigzag

3 day streakD

Protobuf by hand: tags, varints and zigzag

Encode a Protobuf message byte by byte. Once you can do it on paper, the format's size, speed and compatibility rules all make sense.

DepthLesson 2 of 522 min

A Protobuf schema assigns each field a number. On the wire, a message is a sequence of fields, each written as a tag followed by a value. Names never appear. The tag packs the field number and a wire type that tells the reader how to find the value's end, even for fields it does not know.

tool_call.protoText
syntax = "proto3";

message ToolCall {
  int64  id        = 1;
  string name      = 2;
  sint32 priority  = 3;
  repeated int32 ranks = 4;
}
Wire typeNameUsed forHow the reader finds the end
0VARINTint32, int64, uint, bool, enum, sintBytes until one with the top bit clear
1I64fixed64, doubleExactly 8 bytes
2LENstring, bytes, nested messages, packed repeatedA varint length, then that many bytes
5I32fixed32, floatExactly 4 bytes

Varints

A varint writes an integer 7 bits at a time, least significant group first. Each byte's top bit is a continuation bit: 1 means another byte follows, 0 means this is the last. Small numbers take one byte; the largest 64-bit numbers take ten.

Worked exampleEncoding field 1 = 150
Tag
(field 1 << 3) | wire type 0 = 8 = 0x08
150 in binary
1001 0110
Split into 7-bit groups, low first
0010110 then 0000001
Add continuation bits
10010110 = 0x96, then 00000001 = 0x01
Message bytes
08 96 01, 3 bytes
Same data as JSON
{"id":150}, 10 bytes

Names cost nothing on the wire because the schema holds them. That is most of Protobuf's size advantage.

Figure 2.1

Encode a field yourself

Type
Tag = (field 1 << 3) | wire type 0 (VARINT) = 8
0800001000
Value as varint · first bit of each byte = “more follows”
96100101100100000001
Protobuf bytes
08 96 01
Protobuf size
3 B
JSON {"id":150}
10 B
Pick a field number, a type and a value. The tag byte, the varint groups with their continuation bits, and the final bytes update live, beside the JSON equivalent. Try −1 as int32 and then as sint32.

Negative numbers and zigzag

Negative int32 values are sign-extended to 64 bits before varint encoding, so −1 becomes ten bytes. Types sint32 and sint64 use zigzag encoding first, mapping small negatives to small positives: 0→0, −1→1, 1→2, −2→3. Then −1 costs one byte. Choose sint for fields that are often negative.

varint.tsTypeScript
export function encodeVarint(value: bigint): number[] {
  const out: number[] = [];
  let v = BigInt.asUintN(64, value);          // negative int32/int64 → two's complement
  do {
    let byte = Number(v & 0x7fn);
    v >>= 7n;
    if (v) byte |= 0x80;                      // continuation bit
    out.push(byte);
  } while (v);
  return out;
}

export const zigzag32 = (n: number) => ((n << 1) ^ (n >> 31)) >>> 0;
export const tag = (field: number, wireType: number) => encodeVarint(BigInt((field << 3) | wireType));

// field 1 = 150 → [0x08, 0x96, 0x01]
console.log([...tag(1, 0), ...encodeVarint(150n)].map((b) => b.toString(16)));

Check your encoder against protoc --decode_raw, which prints field numbers and values from raw bytes without a schema.

Is Protobuf always faster than JSON?

No. Three things narrow the gap. Compression: gzip removes JSON's repeated names well. Runtime: JavaScript engines parse JSON in highly optimised native code, while a Protobuf library may run in JavaScript. Payload shape: large strings cost the same in both. Benchmark your real payload in your real language before switching.

Go deeperField numbers 1 to 15 and packed repeated fields

The tag is itself a varint. With 3 bits for the wire type, field numbers 1 to 15 fit in one tag byte; 16 to 2,047 need two. Put frequently set fields in 1 to 15.

A repeated int32 field could repeat its tag for every element. In proto3, numeric repeated fields are packed by default: one tag with wire type LEN, a total byte length, then the varints back to back. A list of 1,000 small numbers drops from about 2,000 bytes to about 1,003.

Where you will meet it

gRPC

Every gRPC message is Protobuf behind a 5-byte frame prefix (Module 6).

Storage

Systems store Protobuf in databases and logs for compact, schema-checked records that old and new code can both read.

Debugging

protoc --decode_raw < message.bin reveals the fields of an unknown message, which is often the fastest way to debug a wire problem.

Words to know

Tag
Field number and wire type packed into one varint.
Varint
A variable-length integer, 7 data bits per byte plus a continuation bit.
Wire type
How a field's value is laid out, so readers can skip unknown fields.
Zigzag
A mapping that turns small negative integers into small positive ones.

Check yourself

Answer out loud before you open each one.

Encode field 2, the string "hi".

Tag (2 << 3) | 2 = 0x12, length 0x02, then 0x68 0x69: 12 02 68 69.

Why does −1 as int32 take 10 bytes?

It is sign-extended to 64 bits, so the varint must carry all 64 bits, 7 at a time.

Is Protobuf always faster than JSON?

No. Language runtime, compression and payload shape decide; benchmark the real case.

Try it

Encode ToolCall { id: 150, name: "grep" } by hand on paper. Then write the bytes with printf '\x08\x96\x01\x12\x04grep' | protoc --decode_raw.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap