A Chatbot's Notes Can Take Up More Memory Than the Chatbot

Every time a chatbot writes a word, it leans on notes it made a moment earlier, and those notes can take up more room on the graphics card than the model itself ^[6].

The notes are called the key-value cache, or KV cache. They are the reason a long conversation costs more to serve than a short one, and the reason a model that can accept a whole book as input is not thereby cheap to run. Here is how they work, in the order a curious person would ask.

What does a model do with your prompt?

It chops the text into tokens, which can be whole words or pieces of words, and then writes its answer one token at a time. First it reads everything you sent. Then it predicts one token, adds it to the text, and predicts the next, and so on until it is done ^[1].

Engineers give these two stages names. Reading the prompt is called prefill. Writing the answer is called decoding ^[1][6].

How does a new word use the old ones?

Through a step called attention. It is what lets the calculation for one token draw on information from the tokens before it ^[1].

Think of a library. Every earlier token has a label on its spine and some content inside. The new token arrives with a question. The model compares the question with the labels, finds the best matches and blends their contents into an answer for this step. In the technical words, the question is a query, the labels are keys and the contents are values ^[1][4].

This happens at every layer of the model, and a layer is one stage in the tall stack of stages the text passes through. At each one the model works out key and value numbers for the tokens ^[1][6].

So what is the cache?

It is a place to put those numbers once they are worked out. Without it, each new token would force the model to recompute the keys and values for the whole text so far. With it, the model looks them up and only computes the newest token's numbers, then adds them to the pile ^[1][6].

Here is the picture to hold. The model's weights are the numbers learned in training, and they are its skills. They are fixed while it answers. The cache is a growing set of notes it keeps beside it while it works, so it need not reread and redo every step.

The notes are built fresh from the current prompt and conversation. They are not a memory of your past chats.

The trade is plain. Caching saves arithmetic and costs memory.

Why do the notes get so big?

Because there is a note for every token, at every layer, in every head. A head is one of several parallel ways a layer looks at the text. The cache's size grows with the number of tokens, the number of layers, the number of key-value heads and the number of bytes used to store each number ^[3][4].

One study did the sum for a model called OPT-6.7B at 16-bit precision, meaning each number is stored in 16 bits, and a bit is a single 1 or 0. One sequence of 2,048 tokens needs about one gigabyte of cache. A batch of eight such sequences needs about eight gigabytes ^[3].

That is roughly half a megabyte for each token, by my arithmetic from their figures.

The notes do not stay small as the text grows. A 2026 survey estimated that a Llama-2 7B model at 128,000 tokens can need about 64 gigabytes of cache in FP16, a number format that stores each value in 16 bits. That is a calculation, not a measurement of every model or setup ^[4].

If the model's own weights are also stored at 16 bits, a model with seven billion parameters takes about 14 gigabytes. So in that calculation the notes for one long conversation are more than four times the size of the model writing them.

And what happens with many users?

The weights are one copy that every request can use. The notes are not shared that way, because a batch of requests needs cache space for each request ^[3].

The KIVI research paper gives the extreme case. A 540-billion-parameter PaLM model serving 512 requests with 2,048-token contexts could need three terabytes for the cache alone, three times the size of the model's parameters ^[6].

That is why memory, and not only arithmetic, so often sets the limit. Long prompts, long answers and many simultaneous users all add notes.

A context window, the amount of text a model can accept at once, is only a ceiling. Filling it has a price in memory.

How do engineers shrink the notes?

There are four main moves. Each fixes a different part of the problem.

Pack the notes into blocks. PagedAttention, the method behind the serving system vLLM, splits the cache into fixed-size blocks. They can sit in different places in GPU memory and be handed out as needed. The original paper says this cuts the space wasted by fragmentation, the way a shelf full of odd gaps wastes room, and lets blocks be shared between requests in some cases ^[5]. Nothing is thrown away. The same notes simply fit better.

Write the notes with fewer bits. This is quantization. Each number is stored in a smaller format, rather like rounding prices to the nearest dollar. vLLM supports FP8 cache formats, which store each value in eight bits, half the size of the 16-bit format. Its documentation says this can shrink the cache, so more tokens fit in memory or more requests can be served ^[2].

The KIVI team reported a two-bit method that cut peak memory use by 2.6 times on Llama-2-7B with little to no accuracy loss in the tasks they tested. They also reported higher throughput, meaning more work finished in the same stretch of time, in their workloads. Those are research results and not a promise for every model ^[6].

Keep only some of the notes. This is eviction. The system permanently discards selected entries, betting that the model will not need them again ^[3].

Move the notes somewhere roomier. Offloading shifts cache data from the graphics card's memory to the computer's main memory, which helps when GPU memory is short ^[7].

What does each fix cost?

None is free, which is the part that matters.

Paging adds bookkeeping. The system must track where every block lives, and that is more machinery to build and keep correct. The paper reports its measured gains for the systems and workloads it tested ^[5].

Quantization risks rounding errors. vLLM's documentation notes that calibration, a step in which the system first checks sample values so it can set how the rounding is done, can improve accuracy.

It adds that some attention layers can be left unquantized ^[2]. In other words, you may have to choose which notes can be written in rough pencil.

Eviction is the sharpest gamble. Once an entry is discarded it is gone, so if the detail from page three of a long document is among them, it cannot be brought back. A 2026 comparative study describes possible accuracy loss from eviction, and reports that the effects and performance depend on the model, the workload and the configuration ^[3].

Offloading trades space for time. Hugging Face's documentation says moving cache data between the card and main memory can reduce generation speed ^[7]. The notes are safe, but each trip to fetch them costs a wait.

So the choice runs between memory, speed, complexity and quality, and no option wins on all four. A serving team picks the loss it can live with.

What is still not known

The studies test limited benchmarks, and how well compression holds up on the long, messy conversations people actually have is still open. The published research does not say which of these strategies any particular commercial chatbot uses, or how that changes with request length and load. The trade-offs will also shift across different GPUs, model designs and serving systems.

What you can conclude is smaller and firmer. When a company promises a longer context window, ask what the notes will cost to keep and where they will live. The answer to that decides who can afford to run the model.

Paging and quantization are the two fixes to follow. Both already have a place in a working serving system, vLLM, and both come with published measurements.