The best open-source LLM gateways and observability tools in 2026
The best open-source LLM gateways and observability tools right now are Langfuse, Anthropic Sandbox Runtime (srt) and garak, out of 19 we have reviewed. Each is ranked by our rating, with its price, our verdict and who it suits.
Updated Oct 24, 2026 · Ranked from our reviews · How we review
- 1
Framework-neutral LLM tracing you can own outright: the free self-hosted build covers nearly everything, and the cloud is priced down to the unit.
Best for: Teams that want framework-neutral tracing and evaluation they can self-host for free or buy as a reasonably priced cloud service.
Price: Free and open source under MIT, except ee/ folders under a commercial licence. Cloud: Hobby free (50k units/month); Core $29/month; Pro $199/month (Teams add-on +$300); Enterprise $2,499/month; overage $8 per 100k units, falling to $6 above 50M.
Read our Langfuse review4.6/5 - 2
Your coding agent can be talked into anything; the operating system can't, and srt puts the operating system in charge.
Best for: Every developer running a coding agent on their own machine, as the first layer of protection.
Price: Free and open source under Apache 2.0.
Read our Anthropic Sandbox Runtime (srt) review4.4/5 - 3
The standard open scanner for model-level weaknesses: run it before launch and after every model swap, but do not mistake it for a test of your agent's tools and permissions.
Best for: Teams that want a repeatable, model-level vulnerability scan of an LLM endpoint before launch and after every model change.
Price: Free and open source under Apache 2.0. You pay only for the model calls a scan makes.
Read our garak review4.3/5 - 4
The default self-hosted gateway for teams that need one API, per-team budgets and fallbacks across many providers, provided they pin versions and treat it as security-critical infrastructure.
Best for: Platform teams that must give many developers governed access to many model providers from their own infrastructure.
Price: Free and open source under MIT, except the enterprise/ folder, which is under a separate commercial licence. Enterprise is custom-priced with a free 30-day trial key.
Read our LiteLLM review4.3/5 - 5
Phoenix has the strongest evaluation loop of the open tracing tools, wrapped in a licence and an owner that legal teams will want to discuss.
Best for: Engineers and data scientists who want standards-based tracing with a strong evaluation loop, starting on a laptop.
Price: Free to self-host under the Elastic License 2.0 (source-available; no hosting it as a service for others). Managed Arize AX: Free (25k spans/month, 15-day retention, 1 GB); Pro $50/month (50k spans, 30 days, 10 GB); Enterprise custom.
Read our Arize Phoenix review4.2/5 - 6
PyRIT automates the slow, escalating conversations that break chatbots, provided someone on the team will write the Python that drives it.
Best for: Security engineers who need to red-team chatbots and agents in depth, including multi-turn escalation and web front ends.
Price: Free and open source under MIT. You pay for the model calls used by attacks and LLM-based scorers.
Read our PyRIT review4.2/5 - 7
A heavyweight agent sandbox built for agents you leave running unattended, not ones you sit and watch.
Best for: Teams running agents unattended or on untrusted code who need per-host and per-method network rules.
Price: Free and open source under Apache 2.0.
Read our NVIDIA OpenShell review4.1/5 - 8
Instrument LLM calls once in standard OpenTelemetry and keep your choice of backend, after first changing two defaults that send full prompts to Traceloop's cloud.
Best for: Teams standardised on OpenTelemetry who want LLM traces in their existing observability stack without a vendor SDK.
Price: Free and open source under Apache 2.0. Traceloop's hosted platform, now owned by ServiceNow: Free up to 50,000 spans a month with 24-hour retention; Enterprise (SOC 2, on-prem) by quote.
Read our OpenLLMetry review4.1/5 - 9
Opik gives away the whole server under Apache 2.0 and sells the cheapest hosted plan, but the self-hosted build has no user accounts and phones home by default.
Best for: Teams that want a permissively licensed, self-hostable tracing and evaluation platform, or a low-cost hosted one.
Price: Free and open source under Apache 2.0. Cloud: Free (25k spans/month, 60-day retention); Pro $19/month (100k spans, then $5 per 100k; 400-day retention add-on $29 per 100k spans); Enterprise custom.
Read our Opik review4.1/5 - 10
Scanners tell you whether an MCP server looks dangerous; ToolHive limits what it can do if it is.
Best for: Platform and security teams that need to run, contain and audit MCP servers across a company.
Price: Open source under Apache 2.0; Stacklok Enterprise adds features and support, priced on request.
Read our ToolHive review4.1/5 - 11
NeMo Guardrails is the broadest open policy layer for agents, and it charges for that breadth in configuration, latency and a changelog full of breaking changes.
Best for: Teams building agents or customer-facing assistants that need a configurable policy layer around every model and tool call.
Price: Free and open source under Apache 2.0. NVIDIA's NemoGuard models and NIM microservices are distributed separately.
Read our NVIDIA NeMo Guardrails review4.0/5 - 12
The broadest catalogue of agent and application vulnerabilities in an open red-teaming library, but every verdict rests on an LLM judge you will need to check.
Best for: Python teams building agents who want OWASP-mapped red-team coverage they can run locally.
Price: Free and open source under Apache 2.0. Confident AI's platform runs from a free tier to $200 and $2,000 a month; its red-teaming module is an Enterprise add-on at custom pricing.
Read our DeepTeam review3.9/5 - 13
A sensible first check on the MCP servers and skills your agents trust, provided you accept sending their descriptions to Snyk and scan untrusted configs in a sandbox.
Best for: Teams that let developers or agents install MCP servers and skills and want a supply-chain check before trusting them.
Price: Free and open source under Apache 2.0; needs a Snyk account and API token. Fleet-wide background scanning reports into Snyk Evo, priced on request.
Read our Snyk Agent Scan review3.8/5 - 14
An ambitious open router for choosing models per request, suited to teams already running vLLM who can absorb a fast-moving project.
Best for: Teams serving several models, often on their own vLLM fleet, with enough varied traffic to justify choosing a model per request.
Price: Free and open source under Apache 2.0. No hosted paid service.
Read our vLLM Semantic Router review3.8/5 - 15
Worth putting in front of your skills repo, as long as you believe its own numbers: it catches the obvious, not everything.
Best for: Teams running an internal skills repository who want a scan gate before anything reaches developers.
Price: Free under Apache 2.0; optional LLM and cloud analysers bill through their own providers.
Read our Cisco Skill Scanner review3.7/5 - 16
A neat, MIT-licensed router that has stood still since Palo Alto Networks bought it; fine to run today, hard to build a roadmap on.
Best for: Teams on edge runtimes, or enterprises already buying Palo Alto security, who want a light routing layer.
Price: Free and open source under MIT. Hosted plans: Developer free (10,000 recorded logs a month, 3-day log retention); Production $49/month (100,000 logs, then $9 per 100,000); Enterprise custom.
Read our Portkey AI Gateway review3.7/5 - 17
Worth turning on if your models already live in Weights & Biases; at $0.10 a megabyte, a hard sell for anyone starting fresh.
Best for: Teams already using Weights & Biases for model training who want LLM traces and evaluations in the same place.
Price: SDK free under Apache 2.0. Hosted: Free (1 GB Weave ingestion/month); Pro from $60/month (1.5 GB, then $0.10/MB); Enterprise custom ($0.20/MB overage listed), with dedicated deployment.
Read our W&B Weave review3.6/5 - 18
A capable model router hiding behind a misleading name, and one that ships listening on your whole network with authentication switched off.
Best for: Hobbyists and students who want to run Claude Code or Codex on free and local models.
Price: Free and open source under AGPL-3.0. Model costs depend on the providers you connect; many have free tiers with rate limits.
Read our Free Claude Code review3.4/5 - 19
A sound idea left without an owner: the Apache 2.0 code still runs, but nobody is merging fixes and the hosted pieces are gone.
Best for: Existing users with deterministic validators who can pin versions and patch it themselves.
Price: Free and open source under Apache 2.0. The hosted Hub registry and free remote inference closed on 25 August 2026.
Read our Guardrails AI review3.2/5