What it is
PyRIT (Python Risk Identification Tool) is Microsoft's open framework for assessing the security and safety of generative AI systems. It is built from modular parts: targets (the system under test), converters that transform attack prompts, scorers that judge responses, and a memory that stores every conversation, score and result in SQLite or Azure SQL.
According to the docs it runs single-turn and multi-turn attacks out of the box, including Crescendo, TAP (tree of attacks with pruning) and Skeleton Key. Targets include OpenAI, Azure, Anthropic, Google and Hugging Face models, custom HTTP or WebSocket endpoints, and web apps driven through Playwright. Scorers can be true/false, Likert scale, classification or custom, powered by an LLM, Azure AI Content Safety or your own code. A Scenario framework runs standard assessments for content harms, psychosocial risks and data leakage across hundreds of objectives.
It can be used as a Python library, through the pyrit_scan and pyrit_shell command-line tools, or through CoPyRIT, a local web interface for human-led red teaming. The repository moved from Azure/PyRIT to microsoft/PyRIT; it is MIT-licensed and version 1.1.0 was released on 4 September 2026.
Our review
PyRIT is the open-source toolkit to reach for when a model's weak point is a conversation rather than a single prompt. Crescendo, TAP and Skeleton Key ship as ready-made attacks: an attacker model steers the dialogue, and a scorer decides when the target has given way. garak's probe library covers more one-shot attacks, but it does not adapt mid-conversation in the way PyRIT does.
The design is that of a red team's working tool. Targets, converters and scorers are interchangeable parts, and every conversation, score and result lands in SQLite, Azure SQL or memory, so findings can be re-scored later with a different judge without re-running the attacks. CoPyRIT, started with pyrit_backend on localhost:8000, adds a browser interface for manual testing, and Playwright targets let PyRIT attack a real web chat front end rather than only an API. The 1.0 release in July 2026, followed by 1.0.1 and then 1.1.0 on 4 September, gave the interfaces a stable line after 0.13 and 0.14 earlier in the year.
The cost is effort and tokens. Endpoints and keys live in ~/.pyrit/.env and memory settings in ~/.pyrit/.pyrit_conf, and each turn of a multi-turn attack bills the target, the attacker model and usually an LLM scorer. The project publishes no estimate of tokens per scenario. LLM judges can misgrade, so hits need a human read before they become bug reports. Azure AI Content Safety scoring and Azure SQL memory favour Microsoft's cloud, though neither is required.
The comparison a buyer should make is with promptfoo, not garak. promptfoo puts red-team tests in a YAML file that a product team runs in CI; PyRIT assumes a dedicated person who will read transcripts and tune attacks. They answer different questions, and a team that ships a chatbot weekly may well need both.
Where it earns its place
An AI red-team lead at a bank testing a customer-facing chatbot before launch
Find multi-turn jailbreaks that a single-prompt scan misses
Point an HTTP or Playwright target at the staging chat, run Crescendo and TAP with a separate attacker model, and keep every transcript in SQLite for the report. The software is free; the bill is model calls on target, attacker and scorer, which the project does not estimate.
A security engineer at a SaaS company whose LLM agent reads customer email
Check whether instructions hidden in untrusted input make the agent leak data
Use converters to encode attack text and run the Scenario framework's data-leakage assessments against the agent's endpoint. LLM judges can misgrade, so review each hit by hand before filing it.
A model-risk manager who must evidence AI red-teaming to an internal review board
Keep a repeatable record of what was tested and what failed
Persistent memory stores each conversation, score and result, so the same assessment can be re-run after a model upgrade and compared. It costs engineer time rather than licences; there is no paid edition or vendor support contract.
Before you commit
- Setup
- pip install pyrit on Python 3.10 to 3.14 (the docs recommend 3.13), then put endpoint keys in ~/.pyrit/.env. No GPU is needed unless you host the target or attacker model yourself; pyrit_scan runs a packaged scenario such as airt.scam from the command line.
- Integrations
- Targets for OpenAI, Azure, Anthropic, Google and Hugging Face models, custom HTTP and WebSocket endpoints, and browser front ends through Playwright. Scorers can be an LLM, Azure AI Content Safety or your own code.
- Data and privacy
- Runs locally under MIT; prompts go only to the targets, attacker and scorer models you configure, and results are kept in SQLite, in memory or in Azure SQL if you choose it. The docs do not mention telemetry, and GitHub lists no published security advisories; vulnerabilities go to Microsoft's MSRC.
- Cost at scale
- No paid edition. The cost is model calls: every turn of a multi-turn attack bills three parties, so a scenario run across hundreds of objectives multiplies spend, and no per-scenario token figures are published.
- Limits
- 137 open issues and pull requests. Tests are defined in Python and dotfiles rather than one declarative file, so CI use takes more work than with promptfoo; Content Safety scoring needs an Azure resource.
What's good
- Built-in multi-turn attacks such as Crescendo and TAP
- Pluggable targets, converters and scorers
- Persistent memory for sharing and re-scoring results
- Web UI for human-led red teaming
What's not
- Requires Python and real configuration effort
- Attacker and scorer models add their own token spend to every attack
- LLM-based scores need human review
- Some extras lean on Azure services
Best for
Security engineers who need to red-team chatbots and agents in depth, including multi-turn escalation and web front ends.
Alternatives
PyRIT: common questions
- Is PyRIT worth it?
- PyRIT automates the slow, escalating conversations that break chatbots, provided someone on the team will write the Python that drives it. We rate it 4.2 out of 5.
- How much does PyRIT cost?
- Free and open source under MIT. You pay for the model calls used by attacks and LLM-based scorers.
- What are the downsides of PyRIT?
- Requires Python and real configuration effort; attacker and scorer models add their own token spend to every attack; lLM-based scores need human review; some extras lean on Azure services.
- Who is PyRIT best for?
- Security engineers who need to red-team chatbots and agents in depth, including multi-turn escalation and web front ends.
- How much does PyRIT cost for a team or in production?
- No paid edition. The cost is model calls: every turn of a multi-turn attack bills three parties, so a scenario run across hundreds of objectives multiplies spend, and no per-scenario token figures are published.
- Is PyRIT safe for company data?
- Runs locally under MIT; prompts go only to the targets, attacker and scorer models you configure, and results are kept in SQLite, in memory or in Azure SQL if you choose it. The docs do not mention telemetry, and GitHub lists no published security advisories; vulnerabilities go to Microsoft's MSRC.
- What are the best PyRIT alternatives?
- The closest alternatives we have reviewed are garak, DeepTeam and promptfoo.
Answers checked Oct 23, 2026.
Make PyRIT? Show this rating on your site
