Length oracles, context compaction and KV caches
Compression leaks information through size, and that broke HTTPS twice. Then the agent view: compaction is lossy compression of context, prompt caching is not compression at all, and inference servers compress their caches.
Size is a side channel
Encryption hides content but not length. If a response contains a secret, such as a session token, and also text the attacker controls, compression makes the length depend on both. When the attacker's guess matches the secret, LZ77 finds a repeat and the output shrinks. By trying guesses and watching encrypted sizes, the attacker recovers the secret one character at a time.
Guess a secret from compressed length
GET /search?q=token=?
Cookie: lang=en; token=7f3a9cGuess 7 compresses 9 bits smaller than the worst guess: it extends a match with the real token. Sizes come from this course's toy LZ77 model.
CRIME (2012) used this against TLS-level compression, which was then disabled everywhere. BREACH (2013) applied it to HTTP response compression, which is still on. Defences: never compress secrets alongside attacker-controlled input; mask tokens differently per response; add random padding; or disable compression for sensitive responses. RFC 9842 repeats the warning for dictionaries.
Context compaction is lossy compression
An agent's context window is the most expensive store in the system. When it fills, harnesses compact: summarise older turns into a shorter text and drop the originals. That is lossy compression, and its quality is measured by whether the agent still succeeds at its task. Verbatim details that looked unimportant, such as an exact error message or file path, are often what the next step needed.
| Technique | What it does | Lossy? | Saves |
|---|---|---|---|
| Compaction | Summarises the transcript | Yes, verbatim detail is lost | Context tokens |
| Tool-result clearing | Drops old, re-fetchable tool output | Recoverable by re-running the tool | Context tokens, with no inference cost |
| Prompt caching | Reuses computed work for an identical prefix | No: nothing is removed | Recomputation, cost and latency, not tokens |
| Byte compression (gzip, zstd) | Shrinks stored or transferred bytes | No | Storage and bandwidth, not context |
KV-cache compression in inference
Inference servers store attention keys and values for every token of every active sequence, the KV cache (Module 13). It is often the limit on how many requests fit on a GPU. Servers compress it: quantising cached values to fewer bits, evicting or merging less important tokens, and paging cold blocks to slower memory. Like compaction, the lossy versions trade a little quality for a lot of capacity.
Go deeperChoosing what to keep when compacting
Good compaction is structured, not a generic summary. Keep decisions and their reasons, open questions, file paths touched, exact error messages and commands that worked. Drop exploration that led nowhere and tool output that can be fetched again.
An alternative is a handoff file: the agent writes its state to disk and a fresh session starts by reading it. Comparing compaction and handoff on the same tasks, by success rate and cost, is part of the capstone.
Where you will meet it
CSRF tokens are masked with a fresh random value on every response, so BREACH cannot accumulate matches across requests.
Production coding agents combine tool-result clearing for old outputs with occasional compaction, and track task success when tuning thresholds.
Serving stacks quantise KV caches to 8 bits or fewer to fit more concurrent sequences on each GPU.
Words to know
- Length oracle
- Learning about secret content from the size of compressed, encrypted output.
- Compaction
- Summarising an agent's transcript to free context.
- KV cache
- Stored attention keys and values for tokens already processed.
- Quantisation
- Storing numbers with fewer bits, trading precision for space.
Check yourself
Answer out loud before you open each one.
How can compression leak a secret?
When attacker-controlled text matches the secret, compressed size shrinks. Observing sizes across guesses reveals the secret.
What is the difference between compaction and prompt caching?
Compaction removes context tokens lossily; caching reuses computation for an unchanged prefix and removes nothing.
Why does compaction hurt cache hit rates?
It rewrites the prompt prefix, so the cached work for the old prefix no longer matches.
Try it
In Node, build a string containing token=s3cr3t plus a guess, compress it with zlib.deflateRawSync for every possible next character of the guess, and see which length is smallest.