Writing code with an AI agent is now common, but fixing security holes with one is a harder and riskier job, because a patch that looks right can quietly break the program or miss the real flaw. CWE-bench v2, published by Collinear AI and Hugging Face, is built to measure exactly that: can an agent patch a genuine security weakness in a real codebase without breaking what already works.
The benchmark is 100 tasks spanning 91 different CWEs, the standard catalogue of software weakness types. Each task is a self-contained environment with a realistic instruction and a deterministic check, and the builders say they hand-reviewed every one. Their quality bar is worth noting: the starter code has to fail and the reference fix has to pass, the original feature must still work after the patch, and the grader is set up to reject a model that simply recites the known upstream fix from memory rather than reasoning out a repair. Across all tasks, 95 of the 100 were solved by at least one model, so the set is hard but not impossible.
The ranking is close at the top. Of 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% over four attempts, just ahead of Claude Opus 5.5 at 67.3% pass@1. Both are models we review. But the cost column is where the real decision lives. DeepSeek V4.1 Flash, which we also review, reaches 57.0% at four cents a run, while Opus costs $1.83 and Argon $3.92 for their extra ten points. For a team running patch agents across a large codebase, a model at roughly a hundredth of the price that gets most of the way there is a serious option, not a compromise.
The practical caveat is in the same numbers. Even the leader fails 19% of the set on every attempt, so the long tail of harder weaknesses is far from solved, and a human still has to review security fixes before they ship. The benchmark reports pass rates, not whether a passed patch is the cleanest or most maintainable one.
Two things keep this honest. The task set, grader and harness are all open, so anyone can run their own model and check the numbers rather than take a vendor's word. And the same environments are tagged as reinforcement-learning settings, which means they can train defensive agents as well as score them. That is the more durable contribution: a shared, inspectable yardstick for a job where a quiet miss is a live vulnerability.