Quantization
Quantization is a technique that stores a model's weights, and sometimes activations, with fewer bits, such as 8 or 4 instead of 16, so the model needs less memory and often runs faster, at some cost in accuracy.
Also called: quantized, 4-bit, Q4_K_M
By Divya Sharma · Checked Nov 15, 2026 · How we review
Quantization is the main reason large models fit on consumer GPUs and laptops. Storing weights in 4 bits instead of 16 cuts their memory to roughly a quarter.
How it works
Each weight is mapped from a 16-bit number to a smaller set of values, with a scale factor stored per block of weights so they can be roughly reconstructed during inference. Post-training methods do this after training, with no retraining needed. GPTQ, published in 2022, quantizes GPT-style models to 3 or 4 bits per weight with what its authors call negligible accuracy loss.
Reading the labels
In the GGUF ecosystem around llama.cpp, names like Q8_0, Q5_K_M and Q4_K_M describe the scheme. The number is roughly the bits per weight. K marks the k-quant family, and S, M or L is a small, medium or large mix that keeps some sensitive layers at higher precision. Q4_K_M is a popular balance of size and quality.
Trade-offs
Lower bits mean more quality loss. 8-bit is usually close to the original, 4-bit is a common sweet spot, and below that quality drops faster, especially for small models. Quantization shrinks the weights but not the KV cache unless that is quantized too. Size your setup with the GPU calculator.
Related terms
- GGUF
- GGUF is a binary file format for storing a model's weights, tokenizer and metadata in one file, built for fast loading and local inference with GGML-based tools such as llama.cpp, often in quantized form.
- VRAM
- VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.
- Numeric precision (FP16, BF16, FP8)
- Numeric precision is the number format used to store a model's weights and run its maths, such as FP32, FP16, BF16, FP8 or INT4. Fewer bits per number means less memory and faster inference, at some risk to accuracy.
- Model parameters
- Model parameters are the numbers a neural network learns in training, mostly weights. The parameter count, written as 7B for seven billion, is the usual shorthand for a model's size.
- Local LLM
- A local LLM is a language model that runs on your own computer, phone or server instead of a provider's cloud API, so prompts and data stay on your hardware and it works without an internet connection.