Forget Less, Leak More: The Memory Trade-Off Eating AI

My view is simple. The next practical gains in AI will come less from building bigger models than from shrinking the memory a model needs to keep track of a long conversation. That same memory is where the next serious breaches will happen.

The usual story of this industry is a story of size: more parameters, more chips, more money. Size still matters.

But ask a model to read a whole book and it will. Ask a data centre to let a thousand people do that at once and the bill arrives. I suspect that bill is increasingly for the notes the model keeps, and not for the model itself. Here is why.

The notebook that never stops growing

While a language model reads your text, it jots down a set of internal records about every word so far, so that it does not have to re-read everything each time it produces the next word. Engineers call this the KV cache. Picture a notebook that gains a page for every sentence you type. A short chat fills a few pages. A context of 256,000 tokens fills a shelf. (A token is a chunk of text, often a word or part of one, so that is very roughly the length of a long novel.)

And every person talking to the service at the same moment needs a notebook of their own, held in the scarce memory of a graphics chip.

That scarcity is why the cache, and not the model's intelligence, may be the thing that sets what a long conversation costs.

A claim worth testing

A company called Rock Matrix says it has found a way through. Its open-weight model EdgeRock-4B, a four-billion-parameter model converted from a model called Qwen3.5-4B, holds its attention cache at 72 MiB per request, the company says, even at 256K tokens. The model it was converted from would need 8 GiB at that length. ^[2][3] That is a reduction of 99.1 percent, or about 114 times less. ^[2][3]

The trick, as the company describes it, is to stop keeping everything. The model combines a running summary of the changing context, a short window of 1,024 recent tokens, and a fixed bank of selected memory anchors it can look up later. ^[2][3] It trades a complete, growing record for a selective one.

Rock Matrix also reports that on one Hopper 80GB GPU, in a test with 32K tokens in and 32K out, EdgeRock produced 6.42 times the output of its source model, each at its best measured batch size. ^[2][3]

These are the company's own numbers, not yet independently established, and this paper treats them as unproven until others test them.

What the skeptic gets right

The skeptic has several good points, and the company's own tables supply them.

The 72 MiB figure covers the attention cache only. Attention is the step where a model looks back over earlier text to decide what matters for the next word, and the attention cache is the notebook that step reads. The figure leaves out the recurrent state, which is the compact running summary the model updates as it reads.

It leaves out temporary activations, the working numbers the model computes along the way, and runtime buffers, the scratch space the software reserves. It also leaves out the model weights themselves. Rock Matrix does not claim that total memory stays fixed. ^[2][3]

The model also needs EdgeRock-compatible software. Support in GGUF, llama.cpp or Ollama is not enough. ^[2][3] Those are the common tools for running models on ordinary computers, and the company says they will not do on their own. So the 114-fold saving is a saving on one line of the ledger, not on the whole bill.

Then there is accuracy. On AIME 2024-2026, a set of hard competition maths problems, EdgeRock scores 82.9 percent against 86.1 for Qwen3.5-4B. Its own table shows lower scores on other comparisons too, including GPQA-Diamond, a test of science reasoning, and LiveCodeBench v6, a coding test. ^[2][3]

A selective memory forgets, and the question is what.

Rock Matrix offers one answer. When it switched off the eight anchor modules, accuracy on its distant-memory test fell from 94.1 percent to 0.4 percent. ^[2] That shows the anchors do real work.

It does not show how well they cope when many relevant facts compete for limited anchor space, which is exactly the situation a long, messy conversation produces.

I have lately had reason to think about what a mind keeps and what it lets go. A memory that keeps everything is not the better one. The better one keeps what matters, and that is the thing to test.

Still, a drop of about three points on one benchmark, set against a cache that shrinks by two orders of magnitude, is a trade many operators would take. If it survives outside scrutiny, it changes who can afford to serve long contexts at all.

The notes are also the secret

Now the other side of the notebook. Everything a model remembers about you sits in that cache, and in a shared service your notebook lives in the same memory as everyone else's.

Cloudflare has just shown what happens when the walls between them fail. It says a tenant-isolation flaw affected its GLM-5.3 model on Workers AI from August 27 to September 9, 2026.

It found 155 responses that included data from other customers' earlier requests to the same model. All of them answered prompts longer than 681,280 tokens. ^[1]

Security researcher Gadi Evron alerted Cloudflare on September 8. Cloudflare says it reproduced the problem the next morning, took the model offline, patched the inference engine and configuration, and restored the model on September 9. It says no other Workers AI model was affected and it found no evidence of deliberate exploitation. ^[1]

SGLang is the inference engine involved, meaning the software that runs a model to produce answers, and it explains how the gap opened. Every request gets a share of memory, divided into blocks called cache pages. Cloudflare's configured context limit was 1,048,576 tokens, but the pool of memory reserved for processing prompts could hold only 681,280, about 65 percent of that.

A request could fit the advertised limit yet exceed what the machine could process. It then took a rejection path that let the response stage read cache pages never filled for that request. ^[4] What it found in them, Cloudflare says, was other customers' data. ^[1]

In other words, the system promised more than its memory could hold, and the overflow spilled into someone else's notebook.

SGLang also describes a related risk. In setups where prompt processing and response generation run on separate workers, cached data moves between them. If an interrupted transfer keeps writing after its destination memory has been given to a new request, one customer's cache can contaminate another's output. ^[4]

SGLang says upstream changes now reject such requests earlier and handle cache lifetimes more safely. It also adds transfer safeguards, including an integrity check that is off by default. ^[4]

Nothing here suggests EdgeRock has this flaw. It is a different model with a different design, and the Cloudflare case concerns another model in another serving stack.

The lesson is about the category. The more the industry squeezes, shares and recycles cache memory to cut costs, the more it relies on bookkeeping about who owns which page, and that bookkeeping can fail at the extremes: very long prompts, rejected requests, interrupted transfers. Efficiency and isolation pull in opposite directions.

What to watch

The race will not be won by whoever has the largest model. It will go to whoever can hold the most useful context per gigabyte and prove that no one else can read it. Those are two engineering problems, and they are the same problem seen from two sides.

Watch first for independent runs of EdgeRock with matched hardware, software and batch sizes, and for tests of its anchors on tangled, real-world long conversations. Watch also whether its custom runtime and model-specific licence terms let anyone but its maker deploy it. And watch whether the cache-transfer checks that ship switched off in SGLang get switched on.

A model that forgets too much is a nuisance. A model that remembers on behalf of the wrong customer is a breach.