Workspace

Choosing a codec: gzip, Brotli, zstd, LZ4 or XZ

3 day streakD

Choosing a codec: gzip, Brotli, zstd, LZ4 or XZ

Every codec trades ratio against speed. Learn the decision rule, compress time plus transfer time plus decompress time across all readers, and apply it to APIs, assets and storage.

DepthLesson 4 of 618 min
CodecHow it worksStrengthUse it for
gzip (DEFLATE)LZ77 with a 32 KB window, HuffmanWorks everywhereCompatibility
BrotliLZ77, Huffman, context modelling, built-in web dictionaryBest on small web textStatic assets
ZstandardLZ77, Huffman for literals, FSE for sequencesWide speed range, trainable dictionariesAPIs, logs, storage, Kafka
LZ4LZ only, no entropy stageExtremely fastIn-memory data, realtime paths
XZ (LZMA)LZ with a range coderHigh ratio, slowArchives, packages
Columnar encodingsDictionary, run-length, delta, bit-packing, then a general codecExploits column structureParquet, time series

The decision rule

Compression is worth it when the time it saves on the network exceeds the time it costs on the CPU. Add them up for everyone involved: compress time (once, or per request), transfer time for each reader, and decompress time for each reader. Compress once and serve millions, and a slow, strong level pays off. Compress per request, and a fast level usually wins.

Figure 4.1

When does compression pay?

Times it is read
  1. None800 ms
  2. LZ4384 ms
  3. gzip -6312 ms
  4. zstd -3 · best243 ms
  5. zstd -19692 ms
  6. Brotli -112.19 s
Compress (shared across reads)TransferDecompress
Set the payload size, the reader's bandwidth and how many times the result is read. Bars show total time for each codec, split into compress, transfer and decompress. The speeds and ratios are illustrative orders of magnitude; measure your own data with zstd -b.
Worked exampleA 2 MB JSON API response, compressed per request
Client bandwidth
20 Mbps = 2.5 MB/s
Uncompressed transfer
2 MB ÷ 2.5 MB/s = 800 ms
Fast codec at ~4× ratio, ~400 MB/s compress
5 ms compress + 200 ms transfer
Strong codec at ~6× ratio, ~5 MB/s compress
400 ms compress + 133 ms transfer

The fast level wins by about 330 ms per request. For a static file compressed once at build time, the strong level wins instead.

What clients accept

HTTP content encoding is negotiated: the client sends Accept-Encoding: gzip, br, zstd, and the server picks one and says so in Content-Encoding. Browsers support gzip and Brotli widely and zstd increasingly. Internal services can use whatever both sides agree on.

benchmark.shShell
# Benchmark zstd levels 1 to 19 on your own data: ratio and speed per level
zstd -b1 -e19 sample.json

# Compare with gzip and brotli at a few levels
for l in 1 6 9; do gzip -c -$l sample.json | wc -c; done
for q in 4 9 11; do brotli -c -q $q sample.json | wc -c; done

Plot ratio against compression speed. The curve has a knee; levels past it cost much more CPU for little extra ratio.

Go deeperWhy LZ4 in one layer and zstd in another

Hot paths want speed. An in-memory cache or a replication stream inside a data centre has bandwidth to spare and latency to protect, so LZ4's gigabytes per second matter more than its modest ratio.

Cold storage wants ratio. Data written once and read rarely, such as archived traces, pays storage costs forever, so a strong zstd level or XZ is worth slow compression. Many systems use both: LZ4 for fresh data, recompressed with zstd as it ages.

Where you will meet it

Static sites

Build pipelines precompress assets with Brotli at the maximum quality, because they compress once and serve millions of times.

APIs

Dynamic API responses typically use gzip or Brotli at low levels, or zstd at levels 1–3, to keep per-request CPU small.

Databases

RocksDB-based stores often use LZ4 for upper levels and zstd for the bottom level, where most data lives.

Words to know

Compression ratio
Original size divided by compressed size.
Content-Encoding
The HTTP header naming how a response body was compressed.
Level
A codec setting trading speed for ratio.

Check yourself

Answer out loud before you open each one.

gzip, Brotli or zstd for a dynamic API?

Argue from CPU per request and client support: a fast level of zstd or Brotli where clients accept it, gzip for compatibility.

Why LZ4 in one layer and zstd in another?

Hot paths want speed; cold storage wants ratio.

When does a slow, strong level pay off?

When data is compressed once and read many times, so compression cost is amortised.

Try it

Run zstd -b1 -e19 on a real tool result and a real log file. Find the knee in each curve and pick a level for an API and for archival storage.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap