Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

PagedAttention

PagedAttention is a method for storing a language model's KV cache in small fixed-size blocks that need not sit next to each other in GPU memory, like pages in an operating system. It cuts wasted memory so a server can run more requests at once.

Also called: paged KV cache

By Divya Sharma · Checked Nov 15, 2026 · How we review

While generating text, a model keeps a KV cache: stored attention keys and values for every token so far. It grows with each new token, and nobody knows in advance how long a response will be. Older servers reserved one contiguous chunk of GPU memory per request, sized for the maximum length. Most of that space went unused, and the gaps between chunks could not be used by anyone else.

How it works

The 2023 vLLM paper borrowed the idea of virtual memory paging from operating systems. The KV cache is split into small blocks. Each request has a block table that maps its logical positions to physical blocks anywhere in memory, and new blocks are handed out only when needed. The attention kernel reads through the table. The authors report near-zero waste in KV cache memory and two to four times the throughput of earlier systems at the same latency.

Because blocks are indirect, requests can also share them. Several samples from the same prompt, or many chats with the same long system prompt, can point at one copy of the shared prefix. This is the basis for prefix caching in servers.

Why it matters

Free memory means a bigger batch, and a bigger batch means more users per GPU. PagedAttention is what lets continuous batching add and drop requests freely. It shipped first in vLLM and similar paged designs are now used by most production model servers.

Limits

It does not shrink the KV cache itself; long contexts still need a lot of VRAM. Techniques such as KV cache quantization address that separately.

Related terms

KV cache
A KV cache stores the attention keys and values a transformer has already computed for earlier tokens, so each new token only needs work for itself. It speeds up generation but takes GPU memory that grows with context length.
Model serving
Model serving is running a trained model as a service that applications call over a network, with software that loads the weights onto GPUs, queues and batches requests, manages memory and returns results.
Continuous batching
Continuous batching is a model-serving technique that adds new requests to the running GPU batch and removes finished ones after every generation step, instead of waiting for a whole batch to complete. It keeps the GPU busy and raises throughput.
VRAM
VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.
Attention mechanism
An attention mechanism lets a neural network decide, for each token it processes, how much weight to give every other token in the input. Self-attention is the core operation inside a transformer.

Sources