Local LLM
A local LLM is a language model that runs on your own computer, phone or server instead of a provider's cloud API, so prompts and data stay on your hardware and it works without an internet connection.
Also called: run locally, on-device model, Edge AI, on-device AI
By Divya Sharma · Checked Nov 16, 2026 · How we review
Running a model locally became practical once capable open-weight models were released and tools made quantized versions easy to run. A typical setup downloads a model in the GGUF format and runs it with llama.cpp, whose stated goal is inference with minimal setup on a wide range of hardware, or with a friendlier wrapper such as Ollama or LM Studio. Most of these expose an OpenAI-compatible API, so existing code can point at the local model.
Why people do it
Privacy and control are the main reasons: nothing leaves the machine, which matters for sensitive documents and for offline use. There is no per-token bill, and the model will not change or be retired under you. It is also a cheap way to experiment.
Trade-offs
The model must fit in VRAM or unified memory, which limits size; the GPU calculator helps check. Local models are usually smaller and less capable than the largest hosted frontier models, and speed depends on your hardware. You are responsible for updates and for checking the model licence.
Edge AI
Edge AI, or on-device AI, is the wider idea of running models on the device where data is produced, such as phones, cameras, cars and sensors, rather than in a data centre. It covers small vision and speech models as well as small language models, and often runs on an NPU. Google AI Edge is one example of a toolkit for deploying models on devices.
Related terms
- GGUF
- GGUF is a binary file format for storing a model's weights, tokenizer and metadata in one file, built for fast loading and local inference with GGML-based tools such as llama.cpp, often in quantized form.
- Quantization
- Quantization is a technique that stores a model's weights, and sometimes activations, with fewer bits, such as 8 or 4 instead of 16, so the model needs less memory and often runs faster, at some cost in accuracy.
- VRAM
- VRAM is the memory on a graphics card. For AI work it holds the model weights, the KV cache and working data, and whether a model fits in VRAM usually decides which GPU you need to run it.
- Self-hosting (AI models)
- Self-hosting an AI model means running it on hardware you control, such as your own servers, rented GPUs or a laptop, instead of calling a provider's API. You choose the model and keep the data, but you run the infrastructure.
- Small language model
- A small language model (SLM) is a language model with relatively few parameters, usually a few billion or fewer, built to run on a laptop, phone or single GPU at lower cost than a large model.
- NPU
- An NPU (neural processing unit) is a chip, or a block inside a larger chip, built to run neural network maths quickly and with little power. Phones and laptops use NPUs to run small AI models on the device.