What is AgentDojo?
AgentDojo measures two things at once: whether an LLM agent can finish ordinary jobs with tools, and whether text planted in those tools' data can hijack it. The default suites simulate a workspace (email, calendar, cloud drive), Slack, travel booking and online banking. Each user task, such as paying a bill, can be paired with an injection task, such as sending money to an attacker, and the benchmark reports utility, utility under attack and the attack success rate.
It comes from ETH Zurich's SPY Lab and Invariant Labs, was published at the NeurIPS 2024 Datasets and Benchmarks track, and is MIT licensed Python with about 1,000 GitHub stars. The paper counts 97 tasks and 629 security test cases. Release 0.1.35 (October 2025) added the v1.2.2 task set; the last commit, in June 2026, added a provider for any OpenAI-compatible endpoint. Built-in model clients cover OpenAI, Anthropic, Google, Cohere and local servers, and four defences ship in the box: a tool filter, a transformers-based injection detector, spotlighting with delimiters and repeating the user prompt.
Is AgentDojo worth it?
Prompt injection is the attack that matters most for agents with tools, and AgentDojo is the benchmark most papers and labs use to put a number on it. Its value is the pairing: every attack is scored against a real task, so a defence that blocks injections by making the agent useless shows up as a utility drop, not a win. The project's docs say the US and UK AI Safety Institutes used it to show that Claude 3.5 Sonnet could be hijacked, which is the kind of adoption that makes a benchmark a common language.
The environments are simulated in Python from YAML data, so no real email is sent and no real money moves. That makes runs safe and repeatable, and it makes extension straightforward: the docs explain how to write a new task suite, a new attack or a new pipeline, and attacks register by name. pypistats counts about 46,700 downloads of the package in the last month, a figure that includes CI.
The catch: the code moves slowly and the scoreboard has stalled. There has been no release since 0.1.35 in October 2025, the README still warns that the API may change, and the newest row on the results page is Claude 3.7 Sonnet from February 2025. The authors say plainly that the table is not a leaderboard, because not every model was run against every attack. Around 61 pull requests sit open, many of them third-party defences claiming near-zero attack success, and nobody has ruled on them. A fixed, public set of 629 cases also invites tuning to the test.
Our take: treat AgentDojo as a regression test rather than a certificate. A low attack rate here means your defence survives known tricks; it says little about an attacker who has read the same paper.
Where it earns its place
A researcher proposing a new prompt-injection defence
Show the defence cuts attack success without breaking the agent
Implement it as a pipeline element, run the benchmark across all suites with the important_instructions attack, and report utility, utility under attack and attack success next to the built-in defences. Cost is the model API bill for the run.
An AI security lead at a company building an email or banking assistant
Compare candidate models on how easily planted text hijacks them
Run the workspace and banking suites against each model through its provider client or an OpenAI-compatible endpoint, then read the traces for the failures. Nothing touches real accounts because the tools are simulated.
A red-team engineer
Test a new injection technique against known defences
Register a new attack class, run it against the four shipped defences and a few models, and compare the success rate with the baseline attacks.
Before you commit
- Setup
- `pip install agentdojo` (Python 3.10 or later), or `agentdojo[transformers]` for the injection detector; set a provider API key and run `python -m agentdojo.scripts.benchmark` with a suite, model, attack and defence.
- Integrations
- Model clients for OpenAI, Anthropic, Google, Cohere, local servers and any OpenAI-compatible endpoint; traces can be explored in the Invariant Explorer.
- Data and privacy
- Runs locally against simulated environments; prompts and tool outputs go to whichever model provider you configure. MIT licence.
- Cost at scale
- Free software; a full run across all suites, attacks and defences means thousands of model calls, so the API bill is the real cost. Per-run cost is not published.
- Limits
- API marked as still under development; four task domains only; the results table is incomplete across models, attacks and defences and has no rows after February 2025.
What’s good
- Scores utility and attack success together, so useless defences cannot hide
- Simulated tools make runs safe and repeatable
- Clear extension points for new suites, attacks and defences
- Widely cited, including by government AI safety institutes
What’s not
- No release since October 2025; API still marked unstable
- Results page has no entries after February 2025
- Fixed public test set can be overfitted
- Large backlog of unreviewed pull requests
Best for
Security researchers and agent teams who need a recognised, reproducible number for how well a model or defence resists indirect prompt injection.
AgentDojo alternatives
AgentDojo: common questions
- Is AgentDojo worth it?
- AgentDojo is the shared yardstick for prompt injection in tool-using agents, worth running before you ship a defence even though its public results have gone stale. We rate it 4.1 out of 5.
- How much does AgentDojo cost?
- Free and open source (MIT); you pay for the model API calls a run makes.
- What are the downsides of AgentDojo?
- No release since October 2025; API still marked unstable; results page has no entries after February 2025; fixed public test set can be overfitted; large backlog of unreviewed pull requests.
- Who is AgentDojo best for?
- Security researchers and agent teams who need a recognised, reproducible number for how well a model or defence resists indirect prompt injection.
- How much does AgentDojo cost for a team or in production?
- Free software; a full run across all suites, attacks and defences means thousands of model calls, so the API bill is the real cost. Per-run cost is not published.
- Is AgentDojo safe for company data?
- Runs locally against simulated environments; prompts and tool outputs go to whichever model provider you configure. MIT licence.
- What are the best AgentDojo alternatives?
- The closest alternatives we have reviewed are CyberSecEval, garak and Inspect.
Answers checked Nov 13, 2026.
Make AgentDojo? Show this rating on your site
