Workspace

Length oracles, context compaction and KV caches

3 day streakD

Length oracles, context compaction and KV caches

Compression leaks information through size, and that broke HTTPS twice. Then the agent view: compaction is lossy compression of context, prompt caching is not compression at all, and inference servers compress their caches.

CurrentLesson 6 of 618 min

Size is a side channel

Encryption hides content but not length. If a response contains a secret, such as a session token, and also text the attacker controls, compression makes the length depend on both. When the attacker's guess matches the secret, LZ77 finds a repeat and the output shrinks. By trying guesses and watching encrypted sizes, the attacker recovers the secret one character at a time.

Figure 6.1

Guess a secret from compressed length

GET /search?q=token=? Cookie: lang=en; token=7f3a9c
Recovered: —

Guess 7 compresses 9 bits smaller than the worst guess: it extends a match with the real token. Sizes come from this course's toy LZ77 model.

The page contains a secret token and echoes an attacker-chosen search term. For each candidate next character, the compressed size is shown. The correct guess extends a match and compresses smaller, revealing one more character per round.

CRIME (2012) used this against TLS-level compression, which was then disabled everywhere. BREACH (2013) applied it to HTTP response compression, which is still on. Defences: never compress secrets alongside attacker-controlled input; mask tokens differently per response; add random padding; or disable compression for sensitive responses. RFC 9842 repeats the warning for dictionaries.

Context compaction is lossy compression

An agent's context window is the most expensive store in the system. When it fills, harnesses compact: summarise older turns into a shorter text and drop the originals. That is lossy compression, and its quality is measured by whether the agent still succeeds at its task. Verbatim details that looked unimportant, such as an exact error message or file path, are often what the next step needed.

TechniqueWhat it doesLossy?Saves
CompactionSummarises the transcriptYes, verbatim detail is lostContext tokens
Tool-result clearingDrops old, re-fetchable tool outputRecoverable by re-running the toolContext tokens, with no inference cost
Prompt cachingReuses computed work for an identical prefixNo: nothing is removedRecomputation, cost and latency, not tokens
Byte compression (gzip, zstd)Shrinks stored or transferred bytesNoStorage and bandwidth, not context

KV-cache compression in inference

Inference servers store attention keys and values for every token of every active sequence, the KV cache (Module 13). It is often the limit on how many requests fit on a GPU. Servers compress it: quantising cached values to fewer bits, evicting or merging less important tokens, and paging cold blocks to slower memory. Like compaction, the lossy versions trade a little quality for a lot of capacity.

Go deeperChoosing what to keep when compacting

Good compaction is structured, not a generic summary. Keep decisions and their reasons, open questions, file paths touched, exact error messages and commands that worked. Drop exploration that led nowhere and tool output that can be fetched again.

An alternative is a handoff file: the agent writes its state to disk and a fresh session starts by reading it. Comparing compaction and handoff on the same tasks, by success rate and cost, is part of the capstone.

Where you will meet it

Web frameworks

CSRF tokens are masked with a fresh random value on every response, so BREACH cannot accumulate matches across requests.

Agent harnesses

Production coding agents combine tool-result clearing for old outputs with occasional compaction, and track task success when tuning thresholds.

Inference servers

Serving stacks quantise KV caches to 8 bits or fewer to fit more concurrent sequences on each GPU.

Words to know

Length oracle
Learning about secret content from the size of compressed, encrypted output.
Compaction
Summarising an agent's transcript to free context.
KV cache
Stored attention keys and values for tokens already processed.
Quantisation
Storing numbers with fewer bits, trading precision for space.

Check yourself

Answer out loud before you open each one.

How can compression leak a secret?

When attacker-controlled text matches the secret, compressed size shrinks. Observing sizes across guesses reveals the secret.

What is the difference between compaction and prompt caching?

Compaction removes context tokens lossily; caching reuses computation for an unchanged prefix and removes nothing.

Why does compaction hurt cache hit rates?

It rewrites the prompt prefix, so the cached work for the old prefix no longer matches.

Try it

In Node, build a string containing token=s3cr3t plus a guess, compress it with zlib.deflateRawSync for every possible next character of the guess, and see which length is smallest.

A good place to start

Explore a pattern

A little guidance

Start with a pattern, read through a problem, and mark it solved when the idea clicks. Save anything you want to revisit.

New tutorials are added one at a time. Your bookmarks and solved checkmarks stay in this browser.

Explore the roadmap