State space model
A state space model (SSM) is a sequence model that carries a fixed-size hidden state forward as it reads each token, so its cost grows linearly with input length. Mamba is the best-known example.
Also called: SSM, Mamba, selective state space model
By Divya Sharma · Checked Nov 15, 2026 · How we review
A transformer compares every token with every other token, so cost rises steeply with long inputs, and its KV cache grows with every token. A state space model works more like a running summary. It keeps a hidden state of fixed size and updates it once per token, so memory stays flat and compute grows in a straight line with length.
Where it came from
The idea comes from control theory, where a state space model describes how a system's state changes in response to inputs. The S4 paper in 2021 made this practical for deep learning on very long sequences. Early SSMs did well on long-range benchmarks but lagged transformers on language.
Mamba
Mamba, published in 2023, closed much of that gap. Its key change is making the SSM's parameters depend on the input, so the model can choose what to keep and what to forget at each token. The authors call this a selective SSM, and they report linear scaling in sequence length and higher inference throughput than transformers.
How it is used today
Pure SSMs are less common than hybrids. Many recent models mix a few attention layers with many SSM or similar linear layers, aiming for most of the quality of attention with lower memory use on long context windows.
Limits
A fixed-size state has to compress the past, so exact recall of a detail far back in a long input can be weaker than with attention. Tooling is also less mature: some serving engines and quantization formats support SSM layers later than transformer layers, so check support before you plan to deploy one.
Related terms
- Transformer (architecture)
- The transformer is a neural network architecture, introduced in 2017, that processes sequences using attention instead of recurrence. It is the basis of nearly all current large language models and many image and audio models.
- Attention mechanism
- An attention mechanism lets a neural network decide, for each token it processes, how much weight to give every other token in the input. Self-attention is the core operation inside a transformer.
- KV cache
- A KV cache stores the attention keys and values a transformer has already computed for earlier tokens, so each new token only needs work for itself. It speeds up generation but takes GPU memory that grows with context length.
- Context window
- A context window is the maximum amount of text, counted in tokens, that a language model can consider at once, covering the system prompt, conversation, documents, tool results and the response it writes.
- Mixture of experts
- Mixture of experts (MoE) is a model design in which each layer holds several expert sub-networks and a router sends each token to only a few of them, so only part of the model runs per token.