Skip to content
ReviewRamble

The weekly brief

AI, reviewed, in your inbox.

  • New reviews of tools, models, MCP servers and repos, with the verdict up front
  • Re-checked prices and ratings when something changes
  • New analysis pieces as they are published

Models, agents, MCP servers and the tools actually worth your time.

One email a week. Unsubscribe with the link in any brief. How we use your email

AI red teaming

AI red teaming is the practice of deliberately attacking a model or AI application before and after release, with human testers or automated tools, to find harmful outputs, jailbreaks, injections and other failures so they can be fixed.

Also called: red team, red teaming

By Divya Sharma · Checked Nov 15, 2026 · How we review

The term comes from military and security exercises where a red team plays the attacker. For language models the target is behaviour: can the system be pushed into giving dangerous instructions, leaking data, insulting users or taking an action it should not?

How it works

Ganguli et al. described early work to red team language models in order to discover, measure and reduce harmful outputs. They found that models trained with reinforcement learning from human feedback (RLHF) became harder to red team as they grew larger.

In practice teams mix three approaches. Human red teamers, often domain experts, write creative attacks. Automated tools generate many variants of known attacks, such as jailbreaks and prompt injections, and score the responses. Open-source options include garak, PyRIT and promptfoo. Findings then feed back into training, guardrails or product changes.

Red teaming vs evaluation

Ordinary LLM evaluation asks how well the system does its job on typical inputs. Red teaming asks how it fails on hostile ones. Both are needed, and good red-team findings often become permanent test cases.

Limits

A red team can show that a problem exists; it cannot prove there are none left. Results depend on the skill and imagination of the testers and on how the model is deployed, so a test of a base model says little about an agent built on top of it with tools attached.

Related terms

Jailbreak (LLM)
A jailbreak is a prompt or technique that gets a language model to produce output its safety training is meant to refuse, such as harmful instructions, by role-play, obfuscation, odd encodings or long adversarial text.
LLM evaluation
LLM evaluation, often called evals, is measuring how well a model or AI application performs a task by running it on a set of test cases and scoring the outputs against defined criteria, by code, by humans or by another model.
Prompt injection
Prompt injection is an attack where text fed to a language model, typed by a user or hidden in a web page, file or tool result, changes the model's behaviour in ways its developer did not intend.
Guardrails (AI)
Guardrails are runtime checks placed around a language model that inspect inputs and outputs and block, rewrite or redirect them, so an application stays on topic, avoids harmful content and follows rules the model alone would not reliably keep.
AI alignment
AI alignment is the work of making an AI system's behaviour match what its users and developers actually intend, so it is helpful, honest and avoids harm, rather than just predicting likely text or optimising a narrow score.

Sources