Choosing a codec: gzip, Brotli, zstd, LZ4 or XZ
Every codec trades ratio against speed. Learn the decision rule, compress time plus transfer time plus decompress time across all readers, and apply it to APIs, assets and storage.
| Codec | How it works | Strength | Use it for |
|---|---|---|---|
| gzip (DEFLATE) | LZ77 with a 32 KB window, Huffman | Works everywhere | Compatibility |
| Brotli | LZ77, Huffman, context modelling, built-in web dictionary | Best on small web text | Static assets |
| Zstandard | LZ77, Huffman for literals, FSE for sequences | Wide speed range, trainable dictionaries | APIs, logs, storage, Kafka |
| LZ4 | LZ only, no entropy stage | Extremely fast | In-memory data, realtime paths |
| XZ (LZMA) | LZ with a range coder | High ratio, slow | Archives, packages |
| Columnar encodings | Dictionary, run-length, delta, bit-packing, then a general codec | Exploits column structure | Parquet, time series |
The decision rule
Compression is worth it when the time it saves on the network exceeds the time it costs on the CPU. Add them up for everyone involved: compress time (once, or per request), transfer time for each reader, and decompress time for each reader. Compress once and serve millions, and a slow, strong level pays off. Compress per request, and a fast level usually wins.
When does compression pay?
- None800 ms
- LZ4384 ms
- gzip -6312 ms
- zstd -3 · best243 ms
- zstd -19692 ms
- Brotli -112.19 s
zstd -b.- Client bandwidth
- 20 Mbps = 2.5 MB/s
- Uncompressed transfer
- 2 MB ÷ 2.5 MB/s = 800 ms
- Fast codec at ~4× ratio, ~400 MB/s compress
- 5 ms compress + 200 ms transfer
- Strong codec at ~6× ratio, ~5 MB/s compress
- 400 ms compress + 133 ms transfer
The fast level wins by about 330 ms per request. For a static file compressed once at build time, the strong level wins instead.
What clients accept
HTTP content encoding is negotiated: the client sends Accept-Encoding: gzip, br, zstd, and the server picks one and says so in Content-Encoding. Browsers support gzip and Brotli widely and zstd increasingly. Internal services can use whatever both sides agree on.
# Benchmark zstd levels 1 to 19 on your own data: ratio and speed per level
zstd -b1 -e19 sample.json
# Compare with gzip and brotli at a few levels
for l in 1 6 9; do gzip -c -$l sample.json | wc -c; done
for q in 4 9 11; do brotli -c -q $q sample.json | wc -c; donePlot ratio against compression speed. The curve has a knee; levels past it cost much more CPU for little extra ratio.
Go deeperWhy LZ4 in one layer and zstd in another
Hot paths want speed. An in-memory cache or a replication stream inside a data centre has bandwidth to spare and latency to protect, so LZ4's gigabytes per second matter more than its modest ratio.
Cold storage wants ratio. Data written once and read rarely, such as archived traces, pays storage costs forever, so a strong zstd level or XZ is worth slow compression. Many systems use both: LZ4 for fresh data, recompressed with zstd as it ages.
Where you will meet it
Build pipelines precompress assets with Brotli at the maximum quality, because they compress once and serve millions of times.
Dynamic API responses typically use gzip or Brotli at low levels, or zstd at levels 1–3, to keep per-request CPU small.
RocksDB-based stores often use LZ4 for upper levels and zstd for the bottom level, where most data lives.
Words to know
- Compression ratio
- Original size divided by compressed size.
- Content-Encoding
- The HTTP header naming how a response body was compressed.
- Level
- A codec setting trading speed for ratio.
Check yourself
Answer out loud before you open each one.
gzip, Brotli or zstd for a dynamic API?
Argue from CPU per request and client support: a fast level of zstd or Brotli where clients accept it, gzip for compatibility.
Why LZ4 in one layer and zstd in another?
Hot paths want speed; cold storage wants ratio.
When does a slow, strong level pay off?
When data is compressed once and read many times, so compression cost is amortised.
Try it
Run zstd -b1 -e19 on a real tool result and a real log file. Find the knee in each curve and pick a level for an API and for archival storage.