JSON, Base64 and JSON Lines in depth
JSON looks trivial until a number loses digits, a binary blob grows by a third, or a log file is one giant array. Learn the grammar's edges and the text formats around it.
JSON has six value types: object, array, string, number, true/false and null. Strings are Unicode with backslash escapes. Objects are unordered collections of name-value pairs. That is the whole grammar, which is why every language can parse it and why its weak spots are always the same ones.
Numbers: the 2^53 cliff
The JSON grammar allows any number of digits, but most parsers store numbers as 64-bit doubles (Module 1). Integers above 2^53 = 9,007,199,254,740,992 lose precision silently. The standard itself warns that interoperable numbers should stay within double range.
- Sent by a Go or Java service
{"id": 1234567890123456789}- Exactly representable as a double?
- No: it needs 61 bits of precision; a double has 53
JSON.parse(...).idin JavaScript- 1234567890123456800
- Sent as a string instead
{"id": "1234567890123456789"}survives every parser
Rule: 64-bit IDs, money and anything that must round-trip exactly travel as strings or integers in minor units.
Base64: binary inside text
JSON strings cannot hold arbitrary bytes. Base64 encodes every 3 bytes as 4 printable characters, each carrying 6 bits. The output is always about 33% larger than the input, and = padding marks a final group shorter than 3 bytes. Images and files sent to model APIs as Base64 pay that size, and the decoded bytes are what count, not the characters.
Watch bytes become Base64
48010010006901101001F01111000018S0001106G10011139n11000048w9F100111119910011001821000001039n11100157510011038m0000102C- Input bytes (UTF-8)
- 6
- Base64
- SGnwn5mC
- Growth
- 8 chars, +33%
=.JSON Lines: one record per line
A log of a million events as one JSON array must be parsed whole before you read the first element. JSON Lines (also called NDJSON) puts one complete JSON value per line. Because JSON escapes newlines inside strings, a raw newline can only be a record boundary. You can append, stream, grep and split such files freely; a truncated file loses only its last line.
import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";
// Stream a multi-GB trace file with constant memory
for await (const line of createInterface({ input: createReadStream("trace.jsonl") })) {
if (!line) continue;
const event = JSON.parse(line);
if (event.type === "tool_call") count(event.name);
}Agent traces, eval results and fine-tuning datasets are usually JSON Lines for exactly these reasons.
Go deeperStreaming JSON parsers and duplicate keys
A streaming or SAX-style parser emits events (start object, key, value, end array) as bytes arrive, without building the whole tree. It lets you process a 2 GB array in constant memory, or show a model's partial tool-call arguments before they finish.
JSON's grammar does not forbid duplicate keys: {"a":1,"a":2} is valid. Parsers disagree about which wins, and attackers have used that disagreement when a validating service and an executing service parse the same document differently. Security-sensitive code should reject duplicates.
Where you will meet it
Twitter/X, Discord and Stripe send large IDs as strings, and Stripe sends money as integer minor units.
BigQuery, Snowflake and most log tools import JSON Lines directly, since each line can be processed in parallel.
Images sent to model APIs as Base64 are about a third larger on the wire than the raw file.
Words to know
- Double
- A 64-bit IEEE 754 floating-point number, JSON's usual numeric type in practice.
- Base64
- An encoding of binary data into 64 printable characters, 6 bits each.
- JSON Lines
- One JSON value per line, newline-delimited.
- Streaming parser
- A parser that emits events as input arrives instead of building a whole tree first.
Check yourself
Answer out loud before you open each one.
Why can a 64-bit integer change when parsed as JSON in JavaScript?
It is stored as a double with 53 bits of precision, so integers above 2^53 are rounded.
How much larger is Base64 than its input?
About 33%: every 3 bytes become 4 characters.
Why is JSON Lines better than one big JSON array for logs?
Each line is independent: you can append, stream and process records without parsing the whole file, and corruption is limited to one line.
Try it
In Node, compare Buffer.from(bytes).toString("base64").length to bytes.length for 1, 2, 3, 100 and 1,000 bytes, and explain the padding.