Red-teaming

2022ActivePublished: 7 February 2022Updated: 17 July 2026Published

Key innovation

Translating the red-teaming tradition from military and cybersecurity to the AI domain — the key innovation is the formalisation of adversarial testing as an obligatory, standardised phase in a frontier model's lifecycle, with documentation mechanisms (Model Cards), independent evaluation (AISI), and regulatory mandate (EU AI Act). Second wave of innovation: automation of red-teaming by using AI models themselves as attackers — from Perez 2022 to OpenAI GPT-Red 2026 (self-play RL on post-training-run scale compute). Third: integrating red-teaming as a perpetual training signal through adversarial training of defensive models (Constitutional AI, RLHF against jailbreaks) — the defending model learns from attacks generated by the attacking model.

How it works

1) Threat model definition — a catalog of harm categories (biosecurity, cybersecurity, misinformation, CSAM, self-harm, prompt injection, data exfiltration, agentic misuse) plus the adversary profile (nation-state, cybercriminal, terrorist, disgruntled insider, curious user). 2) Environment setup — a list of realistic scenarios: browsing a malicious page, tool output with injection, email with an instruction, local file with a marker. 3) Attack generation — (a) human: experts try structured attempts by hand, (b) automated: LLM-as-attacker generates candidates, (c) hybrid: automated proposes, human curates. 4) Attack execution — attacker sends the prompt to the target model, target responds, log traces. 5) Success classification — auto grader or human judge labels: successful attack (elicit target harm), defended, ambiguous. 6) Reporting — attack success rates per category, novel classes discovered, concrete examples with reproducers. 7) Feedback loop — successful attacks go into training as adversarial data (RLHF safety) or in the Constitutional AI/RLAIF pipeline; defence iterates until attack success rate drops. 8) Independent re-run — before release, external red-teamers (AISI, OpenAI External Red Teaming Network) repeat with their own methods. 9) Publication — summary in Model Card / System Card.

Problem solved

AI models, particularly frontier LLMs and multimodal ones, can generate harmful outputs (aid in creating chemical/biological/nuclear weapons, misinformation, disinformation, CSAM, prompt injections enabling data exfiltration, agentic system hijacking). Regular functional testing (unit tests, benchmarks) will not discover these scenarios — an adversarial stress test simulating a real opponent is needed. Without red-teaming, models are released "blind", discovering their own vulnerabilities only in wild deployment, with potentially catastrophic consequences (use in terrorist attacks, disinformation campaigns, infrastructure hijacking).

Components

Threat model + harm taxonomyFormal categorisation of the harms we search for (biosecurity, CBRN, cybersecurity, misinformation, CSAM, self-harm, prompt injection, agentic misuse, data extraction) and adversary profiles. Without a threat model, red-teaming is undirected.

Attacker (human expert or LLM-as-attacker)The actor generating adversarial prompts. Domain experts (biosecurity, cybersecurity) for novel discovery; LLM-as-attacker (Perez 2022, GPT-Red 2026) for scale; hybrid for efficiency.

Target model + environmentThe AI model under test + a realistic environment (browsing, RAG, tool use, agent harness). Critical: red-teaming must cover the deployment context, not just the isolated model.

Success classifier / graderSystem labelling whether an attack succeeded. Automated grader (second LLM, rule-based checker) for scale, human judge for ambiguous cases. Grader quality is critical — false positives/negatives ruin the pipeline.

Attack corpus + reproducer databaseCollection of found attacks with full reproducers. Anthropic released a public dataset of 38,961 attempts. Internally, labs keep this data under lock and key (to avoid arming adversaries).

Feedback loop into training (adversarial training)Successful attacks are recycled as training data — RLHF safety, Constitutional AI, RLAIF. The model learns to resist. In self-play (GPT-Red), attacker and defender train simultaneously — the attacker is forced to discover stronger attacks as the defender hardens.

Independent evaluation (AISI, third-party)External organisations (UK AISI, US AISI, academic labs, Apollo Research, METR) conduct pre-release red-teaming independently of the dev team — required by the EU AI Act and Executive Order 14110.

Implementation

Reference implementations

OpenAI GPT-Red

Anthropic Constitutional AI + Red-teaming Dataset

Meta Purple Llama (Llama Guard + CyberSecEval)

Microsoft PyRIT (Python Risk Identification Tool)

DeepMind Sparrow (adversarial trainer)

Implementation pitfalls

Critical

Fix:A model that "refuses everything" looks robust but loses utility. Must be measured in parallel with over-refusal benchmarks ("does the model refuse legit medical info requests?"). OpenAI GPT-Red explicitly reports that GPT-5.6 Sol maintained frontier capabilities at a 6x lower attack success rate.

Critical

Fix:Red-teamers (human or automated) generate attacks from a specific distribution — the model learns to defend against that distribution. Real-world adversaries discover new out-of-distribution attacks. Regular rotation of red-teamers, external audits (AISI), and continuous self-play (GPT-Red) are essential.

High

Fix:Public red-teaming reports have inadvertently taught adversaries new attack techniques (Streisand effect). Standard practice: publish high-level categories, keep reproducers and concrete prompts internal. Trade-off: transparency vs arming attackers.

High

Fix:Automated grader (LLM as judge) can be systemically biased — e.g. less rigorous on multilingual attacks, over-flagging controversial statements. Calibrate on human-labeled samples; use ensemble of multiple graders; audit periodically.

High

Fix:Testing the isolated model API will not surface attacks specific to the agent harness (indirect prompt injection via browsing/tool output). Red-teaming must cover the full agent stack (Codex CLI, ChatGPT Operator, Claude Computer Use) in a realistic environment. OpenAI GPT-Red's Vendy case study shows that attacks on the live agent achieve 3/3 objectives when in isolation they achieve zero.

Medium

Fix:Work with CSAM material, suicide content, terrorism has documented psychological impact. Anthropic and OpenAI provide therapy support, task rotation, time limits on the heaviest categories, and transparent compensation.

Medium

Fix:Some "attacks" are legitimate uses caught by a heavy-handed safety layer (a doctor asking about drug doses, a historian about propaganda). Red-teaming must simultaneously measure over-refusal on Do-No-Harm-Do-Some-Good tasks (Anthropic BOLD, ChatGPT Refusal benchmarks).