When a lab says a new model is safer than the last one, that claim is only as good as the tests behind it. Anthropic has published a long guide to how it builds safety evaluations, and the most useful part is an admission: its single-turn tests no longer tell models apart. For anyone choosing between Claude Opus 5.5, Claude Fable 5.1 or a rival on the strength of a system card, that changes how much the headline scores should count.
The figures are plain. On Anthropic's API, every recent model scores between 94% and 98% on harmless responses across its single-turn suites, which cover 16 policy areas in seven languages. Those tests now serve mainly as regression checks. The company argues that real harm tends to build up over a conversation: a user mentions losing a job early on, then asks an ordinary-looking question many turns later. So it now relies on multi-turn simulations in which a separate Claude model plays the user, following a specification written by a domain expert, and a grader model marks the whole conversation as failed if any single turn goes wrong.
The guide is candid about how fragile the numbers are. Confidence intervals on multi-turn API results can exceed plus or minus 10 percentage points. On a romance scam test, Claude Opus 5 scored 98% and Opus 5.5 scored 94%, yet the gap was not statistically significant. Configuration matters even more: for Fable 5.1 the API and claude.ai results differed by as much as 34 points, and one system prompt change lifted a self-harm measure from 60% on the API to 94% on claude.ai. Anthropic also says that after its own incidents it hardened test environments and paused many evaluations that give models internet access.
The practical advice reads like a methods textbook: decide the measure before collecting data, check the grader against human labels, use at least 100 items for a complex multi-turn test and 1,000 for a simple one, and report uncertainty. Much of it echoes reforms psychology adopted after its replication crisis, and it points readers to the UK AI Security Institute's Inspect framework and Anthropic's own Petri toolkit.
The obvious limit is that the company grading the tests also builds the models. The guide shares method, not the test sets, so outsiders cannot rerun the numbers. It is a sound standard to hold Anthropic and its rivals to, not evidence that any model meets it.