Refusal
How it works
Refusal is instilled during post-training: human preferences teach the model to reject requests judged harmful (RLHF โ Bai et al., 2022), complemented by safety fine-tuning and principle-based methods (Constitutional AI). At inference time the model recognises a request as policy-violating and generates a refusal response instead of substantive content. Mechanistic analysis (Arditi et al., 2024) across 13 open-weight models of up to 72B parameters showed that this behaviour is represented by a single direction in the residual stream (the refusal direction): ablating the direction removes refusals, while adding it induces refusals even for harmless prompts. Adversarial suffixes used in jailbreaks suppress this direction, which the authors read as evidence that current safety fine-tuning is fragile.
Problem solved
Lets a model decline harmful, dangerous, or policy-violating requests without a rigid rule-based input filter, limiting misuse and potential harm while preserving the model's general usefulness.
Components
Human preferences and safety data teach the model to reject requests judged harmful, rather than encoding an explicit rule.
Per Arditi et al. (2024), ablating the direction disables refusals while amplifying it induces refusals even on harmless prompts.
Evolution
RLHF trains the assistant to refuse harmful requests โ the basis of refusal as a learned behaviour (Bai et al., Anthropic).
A suite of 250 safe and 200 unsafe prompts exposing systematic over-refusal (NAACL 2024).
Arditi et al. show a refusal direction in the residual stream and a white-box jailbreak that disables refusals.
The first large-scale over-refusal benchmark (~80k prompts), measured across 32 models from 8 families (ICML 2025).