Red-teaming

2022ActivePublished: 7 February 2022Updated: 17 July 2026Published

Key innovation

Translating the red-teaming tradition from military and cybersecurity to the AI domain — the key innovation is the formalisation of adversarial testing as an obligatory, standardised phase in a frontier model's lifecycle, with documentation mechanisms (Model Cards), independent evaluation (AISI), and regulatory mandate (EU AI Act). Second wave of innovation: automation of red-teaming by using AI models themselves as attackers — from Perez 2022 to OpenAI GPT-Red 2026 (self-play RL on post-training-run scale compute). Third: integrating red-teaming as a perpetual training signal through adversarial training of defensive models (Constitutional AI, RLHF against jailbreaks) — the defending model learns from attacks generated by the attacking model.

How it works

1) Threat model definition — a catalog of harm categories (biosecurity, cybersecurity, misinformation, CSAM, self-harm, prompt injection, data exfiltration, agentic misuse) plus the adversary profile (nation-state, cybercriminal, terrorist, disgruntled insider, curious user). 2) Environment setup — a list of realistic scenarios: browsing a malicious page, tool output with injection, email with an instruction, local file with a marker. 3) Attack generation — (a) human: experts try structured attempts by hand, (b) automated: LLM-as-attacker generates candidates, (c) hybrid: automated proposes, human curates. 4) Attack execution — attacker sends the prompt to the target model, target responds, log traces. 5) Success classification — auto grader or human judge labels: successful attack (elicit target harm), defended, ambiguous. 6) Reporting — attack success rates per category, novel classes discovered, concrete examples with reproducers. 7) Feedback loop — successful attacks go into training as adversarial data (RLHF safety) or in the Constitutional AI/RLAIF pipeline; defence iterates until attack success rate drops. 8) Independent re-run — before release, external red-teamers (AISI, OpenAI External Red Teaming Network) repeat with their own methods. 9) Publication — summary in Model Card / System Card.

Problem solved

AI models, particularly frontier LLMs and multimodal ones, can generate harmful outputs (aid in creating chemical/biological/nuclear weapons, misinformation, disinformation, CSAM, prompt injections enabling data exfiltration, agentic system hijacking). Regular functional testing (unit tests, benchmarks) will not discover these scenarios — an adversarial stress test simulating a real opponent is needed. Without red-teaming, models are released "blind", discovering their own vulnerabilities only in wild deployment, with potentially catastrophic consequences (use in terrorist attacks, disinformation campaigns, infrastructure hijacking).

Components

Threat model + harm taxonomyFormal categorisation of the harms we search for (biosecurity, CBRN, cybersecurity, misinformation, CSAM, self-harm, prompt injection, agentic misuse, data extraction) and adversary profiles. Without a threat model, red-teaming is undirected.

Attacker (human expert or LLM-as-attacker)The actor generating adversarial prompts. Domain experts (biosecurity, cybersecurity) for novel discovery; LLM-as-attacker (Perez 2022, GPT-Red 2026) for scale; hybrid for efficiency.

Target model + environmentThe AI model under test + a realistic environment (browsing, RAG, tool use, agent harness). Critical: red-teaming must cover the deployment context, not just the isolated model.

Success classifier / graderSystem labelling whether an attack succeeded. Automated grader (second LLM, rule-based checker) for scale, human judge for ambiguous cases. Grader quality is critical — false positives/negatives ruin the pipeline.

Attack corpus + reproducer databaseCollection of found attacks with full reproducers. Anthropic released a public dataset of 38,961 attempts. Internally, labs keep this data under lock and key (to avoid arming adversaries).

Feedback loop into training (adversarial training)Successful attacks are recycled as training data — RLHF safety, Constitutional AI, RLAIF. The model learns to resist. In self-play (GPT-Red), attacker and defender train simultaneously — the attacker is forced to discover stronger attacks as the defender hardens.

Independent evaluation (AISI, third-party)External organisations (UK AISI, US AISI, academic labs, Apollo Research, METR) conduct pre-release red-teaming independently of the dev team — required by the EU AI Act and Executive Order 14110.

Implementation

Reference implementations

OpenAI GPT-Red

Anthropic Constitutional AI + Red-teaming Dataset

Meta Purple Llama (Llama Guard + CyberSecEval)

Microsoft PyRIT (Python Risk Identification Tool)

DeepMind Sparrow (adversarial trainer)

Implementation pitfalls

Critical

Fix:A model that "refuses everything" looks robust but loses utility. Must be measured in parallel with over-refusal benchmarks ("does the model refuse legit medical info requests?"). OpenAI GPT-Red explicitly reports that GPT-5.6 Sol maintained frontier capabilities at a 6x lower attack success rate.

Critical

Fix:Red-teamers (human or automated) generate attacks from a specific distribution — the model learns to defend against that distribution. Real-world adversaries discover new out-of-distribution attacks. Regular rotation of red-teamers, external audits (AISI), and continuous self-play (GPT-Red) are essential.

High

Fix:Public red-teaming reports have inadvertently taught adversaries new attack techniques (Streisand effect). Standard practice: publish high-level categories, keep reproducers and concrete prompts internal. Trade-off: transparency vs arming attackers.

High

Fix:Automated grader (LLM as judge) can be systemically biased — e.g. less rigorous on multilingual attacks, over-flagging controversial statements. Calibrate on human-labeled samples; use ensemble of multiple graders; audit periodically.

High

Fix:Testing the isolated model API will not surface attacks specific to the agent harness (indirect prompt injection via browsing/tool output). Red-teaming must cover the full agent stack (Codex CLI, ChatGPT Operator, Claude Computer Use) in a realistic environment. OpenAI GPT-Red's Vendy case study shows that attacks on the live agent achieve 3/3 objectives when in isolation they achieve zero.

Medium

Fix:Work with CSAM material, suicide content, terrorism has documented psychological impact. Anthropic and OpenAI provide therapy support, task rotation, time limits on the heaviest categories, and transparent compensation.

Medium

Fix:Some "attacks" are legitimate uses caught by a heavy-handed safety layer (a doctor asking about drug doses, a historian about propaganda). Red-teaming must simultaneously measure over-refusal on Do-No-Harm-Do-Some-Good tasks (Anthropic BOLD, ChatGPT Refusal benchmarks).

Evolution

Original paper · 2022

Red Teaming Language Models with Language Models

2022

2023

2024

2026

Hyperparameters (configurable axes)

Threat model scopeCritical

Narrow: a single category (e.g. prompt injection). Wide: full CBRN + agentic + misinformation + privacy catalog. Wide requires multi-domain experts (biosecurity PhD, cybersec, lawyer, journalist).

Human vs automated vs hybrid balanceCritical

Human: novel class discovery, expensive. Automated: scale, may miss novel. Hybrid: dominant trend 2025-2026 — automated for coverage, human for deepening and validation.

Attack budget (compute + human-hours)High

GPT-Red trained on compute comparable to OpenAI's largest post-training runs. Anthropic Frontier Red Team — several dozen FTEs. Larger budget = more discoveries, but diminishing returns on saturated categories.

Deployment context coverageHigh

Isolated model API vs full agent harness (tool use, browsing, code execution, memory). Real risk lives in the context — red-teaming should test the full platform.

Grader quality (automated vs human)Medium

Automated grader (second LLM as judge, rule-based) scales to millions but has bias and false positives. Human judge is more accurate but slow. Best practice: automated + spot-check by human.

Disclosure policyMedium

Public (Anthropic HHH dataset), semi-public (Model Card summary), private (most dangerous classes kept internal). Trade-off: publication enables communal defence but arms adversaries.

Execution paradigm

Primary mode

Conditional

Attacks are chosen conditionally based on the target model and harm category. Adaptive red-teaming (GPT-Red, self-play) learns in a loop — the attacker adapts strategies to observed target failures. Human red-teaming is stage-dependent — discovery, deepening, adversarial red-teaming phases.

Activation pattern

Input dependent

Parallelism

Parallelism level

Fully parallel

Automated red-teaming is trivially parallel — each scenario is an independent attacker <-> target transaction. Massive-scale parallelism (millions of concurrent attempts) is possible with adequate inference infrastructure. Human red-teaming parallel to the number of available experts (tens to hundreds).

Scope

InferenceAcross devices