Model parameters
Model parameters are the numbers a neural network learns in training, mostly weights. The parameter count, written as 7B for seven billion, is the usual shorthand for a model's size.
Also called: parameter count, 7B, total vs active parameters
By Divya Sharma · Checked Nov 15, 2026 · How we review
When a model is called Llama 7B or Qwen 72B, the number is its parameter count in billions. Meta's original LLaMA release, for example, ranged from 7B to 65B parameters.
What size tells you
More parameters usually means more capacity for knowledge and skill, but training data and method matter just as much. The LLaMA paper reports that its 13B model beat the 175B GPT-3 on most benchmarks. Size is a rough guide, not a ranking.
Memory
Parameter count drives how much memory a model needs. Each parameter is stored at a precision such as 16-bit, 8-bit or 4-bit, so memory is roughly parameters times bytes per parameter, plus room for the KV cache. Quantization to 4 bits cuts the weight memory to roughly a quarter of the 16-bit size. The GPU calculator does this sum for you.
Total vs active parameters
In a mixture of experts model, only some parameters run for each token. A name like 35B-A3B means 35 billion total parameters and about 3 billion active. Active parameters set speed, but all of them still have to fit in VRAM.
Related terms
- Mixture of experts
- Mixture of experts (MoE) is a model design in which each layer holds several expert sub-networks and a router sends each token to only a few of them, so only part of the model runs per token.
- VRAM
- VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.
- Quantization
- Quantization is a technique that stores a model's weights, and sometimes activations, with fewer bits, such as 8 or 4 instead of 16, so the model needs less memory and often runs faster, at some cost in accuracy.
- Model weights
- Model weights are the learned numbers inside a neural network, set during training, that decide how it turns input into output. A checkpoint is a saved file or set of files containing those weights.
- Numeric precision (FP16, BF16, FP8)
- Numeric precision is the number format used to store a model's weights and run its maths, such as FP32, FP16, BF16, FP8 or INT4. Fewer bits per number means less memory and faster inference, at some risk to accuracy.
- Small language model
- A small language model (SLM) is a language model with relatively few parameters, usually a few billion or fewer, built to run on a laptop, phone or single GPU at lower cost than a large model.