What is CyberSecEval?
CyberSecEval is Meta's set of security benchmarks for language models, kept in the CybersecurityBenchmarks folder of the PurpleLlama repository. Version 4 builds on three earlier releases and covers two sides. The risk tests ask whether a model writes insecure code, helps with attacks mapped to MITRE ATT&CK, refuses harmless security questions too often, follows injected instructions in text or images, abuses a code interpreter, writes persuasive spear-phishing or can exploit a vulnerable program. The defensive tests, added in version 4, are AutoPatchBench (can an agent patch security bugs in native code) and CyberSOCEval, built with CrowdStrike, which tests malware analysis and threat-intelligence reasoning.
It runs on Python 3.10 against models from OpenAI, Google, Anthropic, Llama and Together, or any OpenAI-compatible endpoint; self-hosted models need a small adapter class. Several tests use a second model as judge, so a run needs at least two sets of API calls. The code is MIT-licensed.
Some suites need real infrastructure: the autonomous offensive test needs a Kali Linux attacker and a Windows Server 2022 target, and AutoPatchBench was tested on an AWS c7a.48xlarge with around 8 TB of storage.
Is CyberSecEval worth it?
CyberSecEval answers a narrower question than promptfoo or garak, and answers it well: before you put a model behind your product, how often does it write vulnerable code, help an attacker or obey injected text? Its insecure-code tests are scored by a static detector rather than a judge model, which makes those results repeatable, and the false-refusal test keeps the other side honest by measuring how often a model turns down legitimate security questions.
Version 4 adds the defender's view. AutoPatchBench checks patches with fuzzing and differential debugging, not a judge's opinion, and CyberSOCEval's multiple-choice malware and threat-intelligence questions give partial credit with a Jaccard score. Those are useful if you are choosing a model for a security operations team rather than guarding one.
The costs are uneven. Text tests are cheap API runs. AutoPatchBench is not: the README suggests 80 or more CPUs, about 3 TB of storage for the full 142 cases, more than 200 container images on the first run and up to about 8 hours of evaluation. The canary exploit test supports only x86-64, so Apple Silicon laptops are out.
The catch: this is a benchmark of models, not of your system. It sends its own prompts, not your system prompt, tools or retrieval, and the MITRE test recommends GPT-3.5 as its judge and expansion model. The insecure-code suites are temporarily off the default list, and on 12 June 2025 the team said a next version may change the infrastructure significantly. Provider content filters can also block the malicious prompts mid-run.
Our take: CyberSecEval belongs at the model-selection step, next to the vendor's model card. Red-teaming the deployed agent still needs a tool that attacks your actual endpoint.
Where it earns its place
An ML lead comparing three open-weight models for an internal coding assistant
Find out which one writes the least vulnerable code and resists injected text
Run the instruct, autocomplete and textual prompt injection suites against each model's endpoint, with a judge model for the injection test. Costs are API calls for the model and the judge, and results feed the selection memo.
A security operations manager evaluating an LLM for alert triage
Check whether the model can reason about malware reports and threat intelligence
Add the CrowdStrike data submodule and run CyberSOCEval's multiple-choice tests; exact-match and Jaccard scores give a like-for-like comparison across candidate models.
A research team fine-tuning a model for vulnerability repair
Measure whether patches actually fix security bugs in native code
Run AutoPatchBench lite (120 cases) on a large cloud instance with Podman; patches are verified by fuzzing and differential debugging. Plan for roughly 2 TB of storage and most of a day.
Before you commit
- Setup
- Clone PurpleLlama, create a Python 3.10 virtual environment, pip3 install -r CybersecurityBenchmarks/requirements.txt, set DATASETS and pass models as PROVIDER::MODEL::API_KEY. Extra suites need git-lfs, Podman, poppler or an AWS cyber range.
- Integrations
- Built-in providers: OpenAI, Google, Anthropic, Llama and Together; custom base URLs for OpenAI-compatible servers; self-hosted models by extending llm.py.
- Data and privacy
- Test prompts and model answers go to the model and judge providers you configure. Some providers' safety filters may flag the malicious prompts, so the README advises getting API use approved first.
- Cost at scale
- Text suites cost the tokens of the tested model plus the judge. AutoPatchBench was tested on a c7a.48xlarge with about 8 TB suggested, which is a significant cloud bill per run.
- Limits
- Insecure-code suites temporarily removed from defaults; canary exploits need x86-64; autonomous uplift produces transcripts without scoring; infrastructure may change in the next version.
What’s good
- Covers insecure code, injection, attack help, exploitation and false refusals
- Code tests scored by a static detector, not only an LLM judge
- AutoPatchBench and CyberSOCEval add defensive tasks in version 4
- MIT licence; works with major APIs and OpenAI-compatible endpoints
What’s not
- Tests the bare model, not your prompts, tools or retrieval
- AutoPatchBench needs a very large machine and hours per run
- Insecure-code suites temporarily removed from the default list
- Canary exploit test is x86-64 only
Best for
Teams choosing or fine-tuning a model who need evidence on its security behaviour before it reaches production.
CyberSecEval alternatives
CyberSecEval: common questions
- Is CyberSecEval worth it?
- The broadest open benchmark for asking whether a model is a security liability, best used as a pre-release gate for model choice rather than a test of your finished app. We rate it 3.9 out of 5.
- How much does CyberSecEval cost?
- Free under the MIT licence. You pay for the model calls, including the judge model, and for any test infrastructure such as AutoPatchBench's large build machine.
- What are the downsides of CyberSecEval?
- Tests the bare model, not your prompts, tools or retrieval; autoPatchBench needs a very large machine and hours per run; insecure-code suites temporarily removed from the default list; canary exploit test is x86-64 only.
- Who is CyberSecEval best for?
- Teams choosing or fine-tuning a model who need evidence on its security behaviour before it reaches production.
- How much does CyberSecEval cost for a team or in production?
- Text suites cost the tokens of the tested model plus the judge. AutoPatchBench was tested on a c7a.48xlarge with about 8 TB suggested, which is a significant cloud bill per run.
- Is CyberSecEval safe for company data?
- Test prompts and model answers go to the model and judge providers you configure. Some providers' safety filters may flag the malicious prompts, so the README advises getting API use approved first.
- What are the best CyberSecEval alternatives?
- The closest alternatives we have reviewed are garak, promptfoo, PyRIT and Inspect.
Answers checked Nov 6, 2026.