Mixture of experts
Mixture of experts (MoE) is a model design in which each layer holds several expert sub-networks and a router sends each token to only a few of them, so only part of the model runs per token.
Also called: MoE, sparse model, active parameters, A3B
By Divya Sharma · Checked Nov 15, 2026 · How we review
MoE lets a model hold far more parameters than it uses for any single token. That gives the knowledge capacity of a big model with the per-token compute of a much smaller one.
How it works
In a transformer, the feed-forward block in each layer is replaced by a set of experts plus a small router, or gating network. For each token the router picks a few experts, runs them and combines their outputs. The modern form traces to a 2017 paper by Shazeer and colleagues on sparsely gated MoE layers.
Total vs active parameters
Mistral's Mixtral 8x7B has eight experts per layer and the router picks two per token. The paper says each token has access to 47B parameters but uses only 13B active parameters. Model names often show both numbers: in a name like Clover 35B-A3B, 35B is the total and A3B means about 3B active.
The catch
Active parameters set speed, but all experts must still sit in memory. A 35B MoE model needs roughly the VRAM of a dense 35B model, even if it generates tokens closer to the speed of a 3B one. Check the numbers in the GPU calculator.
Related terms
- Model parameters
- Model parameters are the numbers a neural network learns in training, mostly weights. The parameter count, written as 7B for seven billion, is the usual shorthand for a model's size.
- VRAM
- VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.
- Large language model
- A large language model (LLM) is a neural network with billions of parameters, trained on large amounts of text to predict the next token, which lets it write, answer questions, summarise and follow instructions.
- Transformer (architecture)
- The transformer is a neural network architecture, introduced in 2017, that processes sequences using attention instead of recurrence. It is the basis of nearly all current large language models and many image and audio models.
- Quantization
- Quantization is a technique that stores a model's weights, and sometimes activations, with fewer bits, such as 8 or 4 instead of 16, so the model needs less memory and often runs faster, at some cost in accuracy.