What it is
llama.cpp runs large language models on ordinary hardware. It is written in C and C++ with few dependencies and supports CPUs, Apple Metal, NVIDIA CUDA, AMD HIP, Vulkan, Intel SYCL, WebGPU and a dozen other backends.
It defines GGUF, the single-file format most local models are shared in, quantises models from 1.5-bit to 8-bit so they fit in less memory, and ships llama-server, an HTTP server with an OpenAI-compatible API. Many better-known apps build on it, including llamafile and, historically, Ollama.
The project still tags builds several times a day, but in October 2026 it also began numbered releases: v0.6.0 on 5 October and v0.7.0 on 15 October. v0.7.0 moved the server's default port from 8080 to 9931.
Our review
llama.cpp is still the engine that decides what runs on a laptop, and its new numbered releases make it easier to depend on. The repository has about 132,100 GitHub stars, build tags keep landing several times a day, and v0.7.0 on 15 October 2026 added support for more than ten model families at once, from K2 Horizon to Nemotron Super 3.5 VL.
Reach is the point. One codebase covers x86 and ARM CPUs, Apple silicon, NVIDIA, AMD, Intel, Ascend and Snapdragon hardware, and GGUF is the default way local models are shared on Hugging Face. Recent work goes straight at the memory wall: the v0.7.0 expert cache keeps mixture-of-experts weights in host RAM and runs them on the GPU from an LRU cache, which the maintainers say speeds decoding by up to 2.2 times when experts are offloaded.
The cost of that pace is churn and exposure. Flags and defaults move; v0.7.0 shifted the server to port 9931 and turned formatted chat output into an array of parts, both of which break scripts. Security needs attention too. GGUF parsing has had a run of heap-overflow advisories, and in March 2026 a critical one (CVE-2026-34159) allowed unauthenticated remote code execution through the RPC backend, which the project itself calls fragile and insecure. The server binds to 127.0.0.1 by default and has no API key until you set one.
For many simultaneous users on data-centre GPUs, vLLM and SGLang batch requests far more efficiently, and for a one-command desktop experience Ollama and llamafile hide the knobs. llama.cpp sits beneath both kinds of user, which is why support landing here usually decides when a new open model becomes practical on consumer hardware.
Where it earns its place
A developer on an Apple silicon laptop
Run a capable coding or chat model offline
Install with the one-line script from llama.app, then `llama serve -hf <repo>` downloads a GGUF and starts an OpenAI-compatible endpoint on localhost that editors and agents can point at. There is no per-token cost; the trade is slower output and a smaller model than a cloud API offers.
A platform engineer at a 200-person firm with data-residency rules
Serve an internal assistant on one or two on-premises GPUs
llama-server with parallel slots, an API key and a reverse proxy in front handles modest concurrency while keeping prompts on the firm's hardware. Under heavy load vLLM's batching gets more out of the same GPUs, so this suits a pilot better than a company-wide rollout.
A mobile team shipping an on-device assistant on Snapdragon phones
Put a small quantised model inside an app
The C/C++ library links straight into the app, with Hexagon and OpenCL (Adreno) backends for Qualcomm chips and 1.5-bit to 8-bit quantisation to fit memory. The MIT licence allows closed-source commercial shipping; budget time for frequent changes to llama.h.
Before you commit
- Setup
- One-line installers from llama.app (shell on macOS and Linux, PowerShell on Windows), prebuilt binaries on the releases page, Docker images or a source build. With a GGUF from Hugging Face a first reply takes minutes, most of it download time.
- Integrations
- llama-server exposes an OpenAI-compatible API and a web UI with a models manager; editors, agents and chat apps connect through that API. Backends include CUDA, Metal, HIP, Vulkan, SYCL, CANN, Hexagon, WebGPU and RPC for splitting a model across machines.
- Data and privacy
- Runs locally, and models download from Hugging Face only when you ask. The server listens on 127.0.0.1:9931 with no API key by default (Docker images keep 8080). The RPC server must never face an open network: CVE-2026-34159, rated critical, allowed unauthenticated code execution through it. MIT licence.
- Cost at scale
- Free under MIT with no paid edition, so hardware is the cost. CPU-only and Apple silicon machines handle small models; on Lambda an A10 (24 GB) is $1.29 an hour and an H100 PCIe $3.29, roughly $940 and $2,400 a month if left running.
- Limits
- About 2,600 open issues and pull requests. Server flags and output formats change between releases, high-concurrency throughput trails vLLM and SGLang, and the project describes the RPC backend as proof-of-concept.
What's good
- Runs on almost any hardware, including CPU-only machines
- GGUF and quantisation tooling are the community standard
- OpenAI-compatible server bound to localhost by default
- New model architectures arrive quickly
What's not
- Flags and defaults change often; the default port moved to 9931 in v0.7.0
- Repeated GGUF-parsing and RPC security advisories in 2025 and 2026
- Not built for high-concurrency GPU serving
Best for
Developers who want direct control over running open models locally, on edge devices or on modest servers.
Alternatives
llama.cpp: common questions
- Is llama.cpp worth it?
- If an open model runs on a laptop, it almost certainly runs through llama.cpp, and nothing else covers as much hardware. We rate it 4.8 out of 5.
- How much does llama.cpp cost?
- Free and open source under the MIT licence. No paid edition.
- What are the downsides of llama.cpp?
- Flags and defaults change often; the default port moved to 9931 in v0.7.0; repeated GGUF-parsing and RPC security advisories in 2025 and 2026; not built for high-concurrency GPU serving.
- Who is llama.cpp best for?
- Developers who want direct control over running open models locally, on edge devices or on modest servers.
- How much does llama.cpp cost for a team or in production?
- Free under MIT with no paid edition, so hardware is the cost. CPU-only and Apple silicon machines handle small models; on Lambda an A10 (24 GB) is $1.29 an hour and an H100 PCIe $3.29, roughly $940 and $2,400 a month if left running.
- Is llama.cpp safe for company data?
- Runs locally, and models download from Hugging Face only when you ask. The server listens on 127.0.0.1:9931 with no API key by default (Docker images keep 8080). The RPC server must never face an open network: CVE-2026-34159, rated critical, allowed unauthenticated code execution through it. MIT licence.
- What are the best llama.cpp alternatives?
- The closest alternatives we have reviewed are Ollama, llamafile, MLX and vLLM.
Answers checked Oct 23, 2026.
Make llama.cpp? Show this rating on your site
