Small language model
A small language model (SLM) is a language model with relatively few parameters, usually a few billion or fewer, built to run on a laptop, phone or single GPU at lower cost than a large model.
Also called: SLM
By Divya Sharma · Checked Nov 15, 2026 · How we review
Small language models trade some breadth of knowledge for speed, low cost and the ability to run on local hardware. They suit tasks like classification, extraction, routing, autocomplete and on-device assistants, where a large language model would be slow or expensive.
How they get good
Small models lean on careful training data. Microsoft's Phi-3 report describes phi-3-mini, a 3.8 billion parameter model trained on 3.3 trillion tokens, that the authors say is small enough to run on a phone. Others are made by knowledge distillation, where a small student model learns from a larger teacher.
Running one
With quantization a small model fits in a few gigabytes of memory, so tools such as Ollama or llama.cpp can run it on a laptop. The GPU calculator shows how much memory a given size needs.
Limits
There is no agreed size boundary between small and large. Small models hold less world knowledge, handle long multi-step reasoning less well and are more sensitive to prompt wording. A common pattern is to send easy requests to a small model and hard ones to a bigger model.
Related terms
- Large language model
- A large language model (LLM) is a neural network with billions of parameters, trained on large amounts of text to predict the next token, which lets it write, answer questions, summarise and follow instructions.
- Knowledge distillation
- Knowledge distillation is a training method in which a smaller student model learns to imitate a larger teacher model's outputs, producing a distilled model that keeps much of the teacher's skill at lower cost.
- Local LLM
- A local LLM is a language model that runs on your own computer, phone or server instead of a provider's cloud API, so prompts and data stay on your hardware and it works without an internet connection.
- Quantization
- Quantization is a technique that stores a model's weights, and sometimes activations, with fewer bits, such as 8 or 4 instead of 16, so the model needs less memory and often runs faster, at some cost in accuracy.
- Model parameters
- Model parameters are the numbers a neural network learns in training, mostly weights. The parameter count, written as 7B for seven billion, is the usual shorthand for a model's size.