Fast Chips, Full Notebooks: Why AI Stalls on Memory, Not Math

A chatbot keeps numerical notes on the text it has processed, and with long conversations or many users those notes can fill a chip's memory before its speed runs out ^[3][4].

The notes have a name, the KV cache, and they sit in the memory of the graphics processors (GPUs) that run the models ^[1][2]. This piece explains what the cache is, why it grows, why it can fill a chip that still has speed to spare, and what engineers do about it.

What is a KV cache?

Start with how a language model writes. It produces text one token at a time, and a token is a piece of text, such as a word or part of a word ^[4]. To choose each new token, the model has to look back over everything that came before it.

Picture a child listening to a long story with a notebook in hand. Each time someone asks "what happens next?", the child could start the story over from the first line. Or the child can jot a small note as each part goes by, and flip to the right page when asked. The notes make answers quick.

They also make the notebook fatter with every new part of the story.

The model's notebook works the same way. When it first reads your prompt, it calculates two sets of numbers for each token, called keys and values, which is where the "K" and "V" come from. It stores them so it can reuse them while generating, rather than recomputing them every time ^[2][4].

These notes are not a transcript. They are numerical records, and no person could read a story off them ^[2][4].

Is that the chatbot remembering me?

Not in the way you might hope or fear. The cache is built for the current prompt and the current reply. The documentation describes it as storing and reusing computations for the tokens in that one sequence, not as a record of a user's past conversations ^[2][4].

When the sequence ends, the notebook has done its job.

Why does the cache get so big?

Because it keeps growing while the model works. Every token processed adds its records. The final size depends on the length of the sequence, the batch size, the structure of the model and the number of bytes used to store each value ^[4].

A batch is a group of requests processed together. It matters because batching is how a data centre makes each answer cheaper. A larger batch can lower the cost per request, but every request in it brings its own notebook, so a larger batch also needs more memory for caches ^[1][4].

So two things push the cache up at once. One user pasting in a very long document makes one notebook enormous. Many users at the same time make many notebooks.

How big is big?

One research example gives a sense of scale. Take a 540-billion-parameter PaLM model, a batch of 512 requests, and a context length of 2,048 tokens, which is the amount of text the model holds in view at once, counted in tokens. The researchers estimated the KV cache at three terabytes ^[4].

That is three times the storage taken by the model's own parameters ^[4]. The notes outweighed the model.

That is an extreme case, and the balance shifts with the workload. A chip has to hold two kinds of things in memory. One is the model weights, the learned numbers the model uses to generate text. The other is the growing cache of every active request.

For short sequences, the weights can be the larger demand. With long sequences and large batches, the cache can become the main memory bottleneck ^[3][4].

Why would memory run out before speed?

Here is the part that surprises people. You might assume the limit on a chip is how fast it can calculate. During generation, it can instead be how fast it can fetch.

To produce each next token, the model reads the stored cache ^[4]. Go back to the child. The child can think in an instant, but each answer means flipping through a notebook that gets thicker all the time.

Past a certain thickness, the flipping takes longer than the thinking.

Researchers describe this memory traffic as a limit on inference speed, meaning how fast a finished model can run to produce answers, as opposed to being trained. They report that the chip's computational cores sometimes sit idle while cache data is loaded ^[4]. Sometimes the arithmetic units are not too slow. They are waiting.

The cache also limits how many people a chip can serve. The PagedAttention paper, from the researchers behind the vLLM serving system, argues that the cache's dynamic size limits how many requests can be batched together ^[1]. Every active request needs its share of GPU memory. When the memory is full, the next user waits, however much arithmetic the chip could still do.

What do engineers do about it?

Three broad approaches appear in the research and in serving software. Each costs something.

Allocate the space more carefully

PagedAttention stores a sequence's cache in blocks that need not sit next to one another in GPU memory ^[1][5]. Think of a notebook whose pages can be filed in any free slot on the shelf, instead of needing one long empty stretch reserved in advance. The paper says this reduces wasted memory and allows more requests to be batched ^[1].

The paper reports that vLLM achieved two to four times the throughput of the systems it was compared with, at similar latency ^[1]. Throughput is how many requests a system completes in a given time, and latency is the wait before an answer arrives. Those are results for the systems and workloads the authors tested, not a promise for every deployment ^[1].

The vLLM team's own post makes its benchmark and deployment claims too, and those are the team's reported results ^[5].

Write the notes in fewer digits

The second approach is quantization. It stores each cached value with fewer bits, so the same notes take less room. The cost is that it can affect speed or output quality.

Hugging Face's documentation describes quantized caches as memory-efficient and warns that quantization can hurt latency in some cases ^[2][4].

The KIVI researchers report that their two-bit method used 2.6 times less peak memory, counting the model weights, for Llama-2-7B, with little to no accuracy loss on their tests ^[4]. They say the saving allowed batches up to four times larger and 2.35 to 3.47 times the throughput on their tested workloads ^[4].

A separate paper, KVQuant, aims at a much larger target. Its title promises context lengths of 10 million tokens through cache quantization, and its premise is that cache memory dominates for long sequences ^[3].

Move some notes off the chip

The third option is to shift cache data from GPU memory into the computer's ordinary memory. That relieves the pressure on the GPU. Hugging Face says it can slightly reduce generation throughput, depending on the workload ^[2]. The notes are safe, but fetching them from further away takes time.

Is any of this a free lunch?

No. Each fix buys memory by spending something else, and which currency a service pays in depends on its models, hardware and users. The published gains come from the authors' own tests on named models, so a live service may see different results.

How a live service's memory divides between model weights, active caches and other system use is not something the research reports. The same goes for how widely offloaded and quantized caches run in production, how much they change the number of users per GPU, and what very low-bit compression costs in accuracy on long or reasoning-heavy tasks.

Why does this matter beyond the data centre?

It changes how you can read claims about AI. A model that takes a longer document, or a service that answers more people at once, may owe as much to a better-managed notebook as to a faster chip. That is the paper's view, and the research above gives the reason for it: a chip can be fast and still be held up, its arithmetic units idle while they wait for the notes.

So when a provider talks about longer context windows or lower prices per request, ask how much of the gain came from managing the cache. It is a fair question, and it is worth asking out loud.