Workspace

JSON, Base64 and JSON Lines in depth

3 day streakD

JSON, Base64 and JSON Lines in depth

JSON looks trivial until a number loses digits, a binary blob grows by a third, or a log file is one giant array. Learn the grammar's edges and the text formats around it.

BasicsLesson 1 of 518 min

JSON has six value types: object, array, string, number, true/false and null. Strings are Unicode with backslash escapes. Objects are unordered collections of name-value pairs. That is the whole grammar, which is why every language can parse it and why its weak spots are always the same ones.

Numbers: the 2^53 cliff

The JSON grammar allows any number of digits, but most parsers store numbers as 64-bit doubles (Module 1). Integers above 2^53 = 9,007,199,254,740,992 lose precision silently. The standard itself warns that interoperable numbers should stay within double range.

Worked exampleA 64-bit ID through JSON.parse
Sent by a Go or Java service
{"id": 1234567890123456789}
Exactly representable as a double?
No: it needs 61 bits of precision; a double has 53
JSON.parse(...).id in JavaScript
1234567890123456800
Sent as a string instead
{"id": "1234567890123456789"} survives every parser

Rule: 64-bit IDs, money and anything that must round-trip exactly travel as strings or integers in minor units.

Base64: binary inside text

JSON strings cannot hold arbitrary bytes. Base64 encodes every 3 bytes as 4 printable characters, each carrying 6 bits. The output is always about 33% larger than the input, and = padding marks a final group shorter than 3 bytes. Images and files sent to model APIs as Base64 pay that size, and the decoded bytes are what count, not the characters.

Figure 1.1

Watch bytes become Base64

48010010006901101001F011110000
01001018S0001106G10011139n11000048w
9F1001111199100110018210000010
10011139n11100157510011038m0000102C
Input bytes (UTF-8)
6
Base64
SGnwn5mC
Growth
8 chars, +33%
Type a few characters. Each 3-byte group is split into four 6-bit values, and each value picks one character from the 64-character alphabet. A short final group is padded with =.

JSON Lines: one record per line

A log of a million events as one JSON array must be parsed whole before you read the first element. JSON Lines (also called NDJSON) puts one complete JSON value per line. Because JSON escapes newlines inside strings, a raw newline can only be a record boundary. You can append, stream, grep and split such files freely; a truncated file loses only its last line.

jsonl.tsTypeScript
import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";

// Stream a multi-GB trace file with constant memory
for await (const line of createInterface({ input: createReadStream("trace.jsonl") })) {
  if (!line) continue;
  const event = JSON.parse(line);
  if (event.type === "tool_call") count(event.name);
}

Agent traces, eval results and fine-tuning datasets are usually JSON Lines for exactly these reasons.

Go deeperStreaming JSON parsers and duplicate keys

A streaming or SAX-style parser emits events (start object, key, value, end array) as bytes arrive, without building the whole tree. It lets you process a 2 GB array in constant memory, or show a model's partial tool-call arguments before they finish.

JSON's grammar does not forbid duplicate keys: {"a":1,"a":2} is valid. Parsers disagree about which wins, and attackers have used that disagreement when a validating service and an executing service parse the same document differently. Security-sensitive code should reject duplicates.

Where you will meet it

API design

Twitter/X, Discord and Stripe send large IDs as strings, and Stripe sends money as integer minor units.

Data pipelines

BigQuery, Snowflake and most log tools import JSON Lines directly, since each line can be processed in parallel.

Model inputs

Images sent to model APIs as Base64 are about a third larger on the wire than the raw file.

Words to know

Double
A 64-bit IEEE 754 floating-point number, JSON's usual numeric type in practice.
Base64
An encoding of binary data into 64 printable characters, 6 bits each.
JSON Lines
One JSON value per line, newline-delimited.
Streaming parser
A parser that emits events as input arrives instead of building a whole tree first.

Check yourself

Answer out loud before you open each one.

Why can a 64-bit integer change when parsed as JSON in JavaScript?

It is stored as a double with 53 bits of precision, so integers above 2^53 are rounded.

How much larger is Base64 than its input?

About 33%: every 3 bytes become 4 characters.

Why is JSON Lines better than one big JSON array for logs?

Each line is independent: you can append, stream and process records without parsing the whole file, and corruption is limited to one line.

Try it

In Node, compare Buffer.from(bytes).toString("base64").length to bytes.length for 1, 2, 3, 100 and 1,000 bytes, and explain the padding.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap