What it is
Inspect is an evaluation framework for large language models and agents, developed by the UK AI Security Institute with Meridian Labs, a non-profit. You write evaluations in Python as tasks made of a dataset, a solver (how the model is prompted or run as an agent) and a scorer.
It supports tool use, multi-turn agents and sandboxed execution in Docker, with Kubernetes, Modal, EC2, Proxmox and others available as extensions. It talks to more than 20 model providers plus local models through Hugging Face, vLLM and SGLang, and ships a log viewer (`inspect view`) and a VS Code extension. A separate collection offers more than 200 ready-made evaluations.
Version 0.3.283 was published on 21 October 2026.
Our review
Inspect is the common harness of the AI safety world rather than a product team's test runner, and its roughly 3,070 GitHub stars badly undercount that. Meridian Labs says it is the standard framework at UK AISI, US CAISI, the EU AI Office and the Japanese and Korean institutes, and at METR, Apollo, Epoch and RAND; that list is self-reported. It also moves fast: six releases between 9 and 21 October 2026, all still on 0.3.x.
The design pays off on hard evaluations. Multi-step agents, tool calls, model-graded scoring and long transcripts are first-class, and the log viewer lets you step through every message of every sample. The 200-plus evals in the companion collection mean a lab can run standard benchmarks and its own tasks through one pipeline.
Sandboxing deserves care. Only `docker` and `local` are built in, and `local` is no sandbox at all: it runs as your user. The auto-generated Docker config sets `network_mode: none`, but supplying your own compose file drops that, and Compose's default network reaches the internet. Writable host mounts are blocked unless you opt in, and output over the 10 MB exec limit is cut from the front without an error, which can silently change what a scorer sees.
For a product team that wants to assert on prompts in CI, promptfoo's config files or DeepEval's pytest style get there with less code. Inspect expects Python, has no hosted dashboard and no paid tier, and its community, though expert, is smaller.
CASRAI's guide makes the point worth keeping: a shared harness makes results from different institutes comparable, but it is infrastructure, not assurance. The task you write inside it still decides what the number means.
Where it earns its place
A safety researcher at an independent evaluation organisation
Run capture-the-flag style agent evaluations against several frontier APIs
Define the task with a Docker sandbox per sample and run it across providers from one script. By default Docker runs up to twice as many sandboxes as the machine has processors, so a large CPU box shortens runs; the spend is API tokens plus that machine.
An ML engineer at a model developer
Compare a fine-tuned open model with an API baseline on standard benchmarks
Serve the open model through vLLM, point Inspect at both, and pull benchmarks from the 200-plus collection. One harness for both models removes prompt-format drift between them; the cost is GPU time for the local model.
An AI governance lead at a bank
Produce evidence for a model-risk review of an internal assistant
Have the data science team encode the bank's test cases as Inspect tasks and archive the logs, which record every prompt, response and score. Free to use, but there is no vendor support contract, so the bank owns the harness.
Before you commit
- Setup
- `pip install inspect-ai` plus the provider SDK (for example `pip install anthropic`) and an API key; a first eval runs from a short Python file with `inspect eval`. Docker is needed for sandboxed agent tasks.
- Integrations
- More than 20 model providers, local inference via Hugging Face, vLLM and SGLang; sandboxes in Docker built in, with k8s, Daytona, Modal, EC2, Proxmox, Vagrant and OpenShell as extension packages; ControlArena and interpretability libraries build on it.
- Data and privacy
- Runs on your machines; prompts go only to the model providers you configure, and logs are written locally. Telemetry is not mentioned in the docs. MIT licence. The local sandbox gives agents your user's access, and custom tools and scorers can reach the internet even when the sandbox cannot.
- Cost at scale
- No paid or hosted edition. Cost is model tokens for every sample and grader call, plus compute for sandboxes, which scales out through the Kubernetes or Modal extensions for large eval sets.
- Limits
- About 460 open issues and pull requests; 0.x versioning with very frequent releases; file reads capped at 100 MB and exec output at 10 MB, truncated silently; no default CPU or memory cap on Docker sandboxes.
What's good
- Built for agents, tool use and model-graded scoring
- Docker sandbox with network off by default, plus k8s and cloud extensions
- Step-through log viewer and VS Code extension
- 200+ ready-made evals; used across government safety institutes
What's not
- Python only, with more setup than CI-oriented tools
- Custom compose files silently re-enable network access
- Still 0.x with releases every few days
- Smaller community than promptfoo or DeepEval
Best for
Researchers and safety teams running agent and capability evaluations.
Alternatives
Inspect: common questions
- Is Inspect worth it?
- Inspect is the shared harness of government and independent AI safety testing; product teams checking prompts in CI will find it more framework than they need. We rate it 4.4 out of 5.
- How much does Inspect cost?
- Free and open source under the MIT license.
- What are the downsides of Inspect?
- Python only, with more setup than CI-oriented tools; custom compose files silently re-enable network access; still 0.x with releases every few days; smaller community than promptfoo or DeepEval.
- Who is Inspect best for?
- Researchers and safety teams running agent and capability evaluations.
- How much does Inspect cost for a team or in production?
- No paid or hosted edition. Cost is model tokens for every sample and grader call, plus compute for sandboxes, which scales out through the Kubernetes or Modal extensions for large eval sets.
- Is Inspect safe for company data?
- Runs on your machines; prompts go only to the model providers you configure, and logs are written locally. Telemetry is not mentioned in the docs. MIT licence. The local sandbox gives agents your user's access, and custom tools and scorers can reach the internet even when the sandbox cannot.
- What are the best Inspect alternatives?
- The closest alternatives we have reviewed are LM Evaluation Harness, promptfoo and DeepEval.
Answers checked Oct 23, 2026.
Make Inspect? Show this rating on your site
