Step 16 of 22
Inference engines
Run models fast and cheaply, on your GPUs or a provider's. Serving decides your latency and your bill. Open models only pay off if you can serve them efficiently.
How to choose
- Production GPUs: a high-throughput server such as vLLM or SGLang.
- Laptops and edge devices: llama.cpp, Ollama or MLX.
- No GPUs to manage: a hosted inference provider, billed per token.
What to look for
The questions that separate the options, in order of how much they matter.
01
Throughput or latency: decide which you are buying
Throughput is total tokens per second across all users, and it sets your cost per token. Latency is what one user feels: time to first token (TTFT) and output speed for their request. Engines such as vLLM and SGLang batch many requests together to raise throughput, which can slow each individual stream; specialist chips (Groq, Cerebras) sell very high per-user speed. A chatbot needs low TTFT, a nightly batch job needs throughput, and an agent making dozens of sequential calls needs both.
02
Reading Artificial Analysis provider benchmarks
Artificial Analysis times every provider serving the same model. Output speed is tokens per second after the first token arrives; TTFT is seconds until the first token, which for reasoning models means the first reasoning token, so also check time to first answer token. Figures are the median over the past 72 hours, the default prompt is about 10,000 input tokens, and tests run from one Google Cloud region in Iowa, so TTFT includes network distance from there. Most tests send one request at a time; the parallel test sends only 10, once a day, so the charts say little about how a provider holds up under your peak load.
03
Hardware support
The engine has to run well on what you own or rent. TensorRT-LLM targets only NVIDIA. vLLM and SGLang cover NVIDIA and AMD, with Intel and Ascend support in places. llama.cpp reaches CPUs, Apple Metal, CUDA, AMD and Vulkan; MLX is for Apple silicon only. Hosted providers decide this for you: Groq runs its own LPU chips, Cerebras its wafer-scale chips, Together AI and Fireworks NVIDIA GPUs.
04
Quantisation formats
Quantisation stores weights in fewer bits so a model fits on fewer GPUs and runs faster, usually at a small quality cost. Formats are tied to hardware: FP8 needs Ada or Hopper GPUs or newer, NVFP4 and MXFP4 need Blackwell for native speed, and GGUF is the format for llama.cpp, Ollama and most laptop use. Check that the engine supports the exact checkpoint you plan to download, and benchmark quality on your own task after quantising. Hosted providers rarely publish the precision they serve, which is one reason the same model can answer differently on two of them.
05
OpenAI-compatible API and its gaps
Almost every option here speaks the OpenAI chat-completions format, so switching is mostly a base URL and model name. The gaps are in the details: Groq rejects logprobs and logit_bias, Cerebras limits n to 1 and on gpt-oss-120b will not combine tools with a response schema, and Together uses namespaced model IDs. Test the exact parameters your code sends, including streaming and tool calls.
06
Structured output and tool calling
Agents and extraction jobs need JSON that matches a schema every time. Engines enforce this by constraining which tokens may be generated; SGLang has made structured generation a core feature, and Together supports response_format on its OpenAI-compatible API. Check whether schema enforcement works together with tool calling and streaming on the model you use, because support differs per model, not just per provider.
07
Many fine-tunes on one base model
If you serve several LoRA fine-tunes, check whether the engine or provider can load many adapters on one copy of the base model and pick the right one per request. Without it, each fine-tune needs its own GPU. Fireworks, for example, serves fine-tuned models at the same price as the base model.
08
Licence and pricing basis
Self-hosted engines are free software, mostly Apache 2.0 or MIT; you pay for GPUs and the people to run them. Hosted providers bill per million tokens with input and output priced separately, and most add batch discounts, cached-input discounts and dedicated hourly GPUs. At steady high volume, self-hosting on rented GPUs usually wins; at spiky or low volume, per-token billing does.
How the options differ
Self-hosted engines are software you run on your own or rented hardware. Server engines (vLLM, SGLang, TensorRT-LLM) are built for data-centre GPUs and many concurrent users: vLLM is the broad default, SGLang excels where requests share long prefixes, and TensorRT-LLM squeezes the most from NVIDIA hardware at the cost of complexity. Local engines (llama.cpp, Ollama, MLX, colibri) are built for one user on a laptop, workstation or edge device: llama.cpp is the portable core, Ollama wraps it for one-command use, MLX targets Apple silicon, and colibri streams experts from disk to run very large mixture-of-experts models on modest machines, slowly.
Hosted inference providers (Groq, Cerebras, Together AI, Fireworks AI) run the models for you and bill per token. Groq and Cerebras build their own chips and lead on raw speed for a short list of models; Together AI and Fireworks run NVIDIA GPUs with broad catalogues, fine-tuning and dedicated endpoints. You trade control of hardware, quantisation and data location for zero operations.
11 tools, compared
Ranked by our rating. Open any review for sources, prices and the full verdict.
| Tool | Rating | Our verdict | Best for | Price |
|---|---|---|---|---|
| llama.cpprepo | 4.8/5 | If an open model runs on a laptop, it almost certainly runs through llama.cpp, and nothing else covers as much hardware. | Developers who want direct control over running open models locally, on edge devices or on modest servers. | Free and open source under the MIT licence. No paid edition. |
| vLLMrepo | 4.7/5 | Still the engine to put an open model behind a multi-user API, provided someone on the team owns its upgrades and its network exposure. | Teams serving open models to many concurrent users on GPU servers. | Free and open source under the Apache 2.0 licence. You pay for the GPUs it runs on. |
| Ollamarepo | 4.6/5 | Local models in one command, with defaults (a 4,096-token context, one request at a time, no login) that suit a laptop and nothing bigger. | Developers and individuals who want open models running locally with minimal setup. | Open source (MIT) and free to run locally. Optional Ollama Cloud: Free with starter credits; Pro $20/month ($60 credits); Max $100/month ($300 credits); Team $500/month ($1,000 shared credits, unlimited users); Enterprise custom. |
| Together AItool | 4.5/5 | A single bill for open models at every stage, from a $15 API top-up to a reserved H100 cluster, at prices that undercut the specialist hosts on dedicated hardware. | Teams building on open models who want to move from API calls to fine-tuning and their own GPUs with one provider. | Prepaid, $15 minimum first purchase, no free trial. Per 1M tokens, e.g. gpt-oss-120B $0.15 in / $0.60 out, Kimi K3 $3 / $15; batch up to 50% off. Dedicated endpoints: H100 $5.49/hour, B200 $8.99/hour. GPU clusters: H100 $3.99/hour on demand, $1.99 preemptible. Fine-tuning from $0.34 per 1M tokens. |
| SGLangrepo | 4.4/5 | SGLang is built for the agent and RL workloads that resend the same long prompts, and it belongs behind a locked-down network. | Teams serving open models with heavy prompt reuse, such as agents and multi-turn chat. | Free and open source under the Apache 2.0 license. |
| MLXrepo | 4.4/5 | MLX turns a high-memory Mac into a credible box for local inference and LoRA tuning; off Apple silicon it is a port, not a reason to switch. | Mac-based developers and researchers who want native speed and Python control on Apple silicon. | Free and open source under the MIT license. |
| Fireworks AItool | 4.4/5 | Fireworks is a safe production home for open models: zero retention by default, real certifications and fine-tuned models at base prices, with dedicated GPUs as the one expensive part. | Teams running open models in production who want fine-tuning and predictable per-token pricing. | Prepaid credits. Per 1M tokens, e.g. GPT OSS 120B $0.15 in / $0.60 out, Kimi K3 $3 / $15. Batch 50% off. Dedicated H100/H200 $8/hour, B200 $13/hour. Fine-tuning from $0.50 per 1M training tokens. |
| colibrirepo | 4.2/5 | A disk-streaming C engine that makes 744B-parameter open models run on a desktop, slowly, with an honest local stand-in for the Jev decision API on the side. | Enthusiasts and developers who want to run frontier-size open MoE models, or prototype Jev-style decisions, on their own machine. | Free and open source (Apache 2.0). Each model keeps its own licence. |
| TensorRT-LLMrepo | 4.0/5 | TensorRT-LLM is where new NVIDIA silicon gets its first serious optimisations, and the price is a long CVE list and a stable release cadence measured in seasons. | Teams running large NVIDIA GPU fleets who want maximum throughput per GPU. | Free and open source under the Apache 2.0 license (some bundled components carry their own licenses). |
| Groqtool | 4.0/5 | Groq still sells some of the fastest tokens on the market at bargain prices, but the company behind it is no longer the chip maker people signed up with, and its public catalogue has shrunk to fit. | Developers building voice, real-time or agent features where response speed matters more than model choice. | Free plan with rate limits (30 requests/minute and 1,000/day for gpt-oss-120b). Pay-as-you-go Developer plan: gpt-oss-120b $0.15 in / $0.60 out per 1M tokens, gpt-oss-20b $0.075 / $0.30, qwen3.8-27b $0.80 / $4.00; Whisper large-v3-turbo $0.04/hour of audio. Llama and some newer models via sales. |
| Cerebrastool | 4.0/5 | Cerebras sells speed by the token, and at more than twice Groq's input price for gpt-oss it only pays off when latency is the product. | Developers building latency-sensitive agents or coding tools that can work with gpt-oss or Qwen models. | $5 one-time trial credit (30 days), then pay-as-you-go. gpt-oss-120b $0.35 in / $0.75 out per 1M tokens; Qwen 3.8 27B $0.99 / $1.49. Dedicated Inference by agreement. Cerebras Code subscriptions ($50 and $200/month) listed as sold out. |
Side by side
The specifications that matter for this step. Checked 2026-10-19; confirm on the vendor’s site before you commit.
| Tool | Type | Hardware | Quantisation formats | OpenAI-compatible API | Licence / pricing basis |
|---|---|---|---|---|---|
| llama.cpp | Engine (local) | CPU, Metal, CUDA, AMD, Vulkan | GGUF | Yes (llama-server) | MIT |
| vLLM | Engine (server) | NVIDIA, AMD, Intel GPU, CPU | FP8, AWQ, GPTQ, GGUF, bitsandbytes | Yes | Apache 2.0 |
| Ollama | Engine (local) | macOS, Windows, Linux | GGUF | Yes | MIT; optional paid cloud |
| Together AI | Hosted provider | NVIDIA GPUs (H100, B200) | Not published | Yes | Per token or per GPU-hour |
| SGLang | Engine (server) | NVIDIA, AMD, Ascend, Intel CPU | FP8, MXFP4, NVFP4, AWQ, GPTQ, GGUF | Yes | Apache 2.0 |
| MLX | Framework (local) | Apple silicon only | MLX quantised (e.g. 4-bit) | Not published | MIT |
| Fireworks AI | Hosted provider | NVIDIA H100, H200, B200 | Not published | Yes | Per token or per GPU-hour |
| colibri | Engine (local, MoE) | CPU; CUDA or Vulkan optional | Not published | Yes, plus Anthropic API | Apache 2.0 |
| TensorRT-LLM | Engine (server) | NVIDIA only | FP8, NVFP4, MXFP4, AWQ, GPTQ | Yes (trtllm-serve) | Apache 2.0 |
| Groq | Hosted provider | Groq LPU | Not published | Mostly (no logprobs) | Per token |
| Cerebras | Hosted provider | Cerebras wafer-scale chips | Not published | Yes, with limits | Per token |
Where this stage fits
- Chat with your documents: we would start with Together AI.
- Voice agent: we would start with Groq.
- Private, on-device AI: we would start with Ollama, llama.cpp, MLX.
- Your own fine-tuned model: we would start with vLLM.
Sources for this guide
- Artificial Analysis: performance benchmarking methodology
- vLLM docs: quantisation support
- SGLang docs: quantisation
- TensorRT-LLM docs: quantisation
- TensorRT-LLM docs: trtllm-serve
- Groq docs: OpenAI compatibility
- Cerebras docs: OpenAI compatibility
- Together AI docs: OpenAI API compatibility
- Fireworks AI docs: OpenAI compatibility