KV cache
A KV cache stores the attention keys and values a transformer has already computed for earlier tokens, so each new token only needs work for itself. It speeds up generation but takes GPU memory that grows with context length.
Also called: key-value cache
By Divya Sharma · Checked Nov 16, 2026 · How we review
In a transformer, every new token attends to every earlier token through the attention mechanism, which uses a key and a value vector per token per layer. Without a cache, the model would recompute those vectors for the whole sequence at every step. The KV cache keeps them in memory, so generating the next token only adds one new set.
The memory cost
The cache grows with the number of tokens in context, the number of layers, and the number of concurrent requests. For long prompts or many users it can take as much VRAM as the weights themselves, which is why a model can fit on a card but fail at long context. The PagedAttention paper describes KV cache memory per request as huge and changing size as requests grow and finish; wasted cache memory limits how many requests a server can batch.
How engines manage it
Paged attention, introduced with vLLM, stores the cache in fixed-size blocks, like virtual memory pages, to avoid fragmentation and to share blocks between requests with the same prefix. Engines may also quantize the cache to lower precision or offload it to CPU memory.
KV cache and prompt caching
The KV cache lives inside one generation. Prompt caching, offered by API providers, keeps the computed state for a repeated prompt prefix across separate requests, which saves time and money. The GPU calculator includes context length in its memory estimate.
Related terms
- VRAM
- VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.
- PagedAttention
- PagedAttention is a method for storing a language model's KV cache in small fixed-size blocks that need not sit next to each other in GPU memory, like pages in an operating system. It cuts wasted memory so a server can run more requests at once.
- Prompt caching
- Prompt caching is an API feature that stores the processed form of a repeated prompt prefix, such as a long system prompt or document, so later requests that start with the same prefix are cheaper and faster.
- Context window
- A context window is the maximum amount of text, counted in tokens, that a language model can consider at once, covering the system prompt, conversation, documents, tool results and the response it writes.
- Attention mechanism
- An attention mechanism lets a neural network decide, for each token it processes, how much weight to give every other token in the input. Self-attention is the core operation inside a transformer.