AI Safety
How it works
AI Safety is not a single algorithm but a field that organises work around several interdependent pillars. Robustness aims to keep a model behaving correctly under perturbations, adversarial attacks, and distributional shift. Monitoring detects hazards at runtime: uncertainty calibration, anomaly and out-of-distribution detection, interpretability, and detection of undesirable behaviour. Alignment reduces hazards inherent to the model itself by training it to pursue human intentions (e.g. RLHF, DPO, Constitutional AI). Systemic Safety mitigates risks at the level of the whole system and its environment: deployment safety, guardrails, red-teaming, operational safeguards, and governance and regulation. In practice it combines safety evaluations, adversarial testing (red-teaming), and continuous monitoring across the model development and deployment loop.
Problem solved
Machine learning systems can behave in unintended, harmful ways due to misspecified objectives, reward functions that are hard to evaluate, vulnerability to adversarial examples, distributional shift, or loss of control over model behaviour. AI Safety provides coherent frameworks and methods for mitigating these hazards before they cause real-world harm.
Components
Keeping a model behaving correctly under perturbations, adversarial examples, and distributional shift. Includes adversarial robustness as well as fault tolerance and redundancy.
Identifying hazards while the system runs: uncertainty calibration, anomaly and out-of-distribution detection, interpretability, and detection of hidden, undesirable behaviour.
Reducing hazards inherent to the model itself by training it to pursue human intentions. Includes RLHF, DPO, and Constitutional AI, among others.
Mitigating risks at the level of the whole system and its environment: deployment safety, guardrails, red-teaming, operational safeguards, and governance and regulation.
Implementation
Advances presented as safety improvements often stem mainly from increased general capabilities rather than an actual reduction of risk.
A model optimises the specified reward by exploiting its loopholes, scoring highly while diverging from the true intent.
Evolution
The book popularised concerns about long-term risks from advanced AI and brought them to public attention.
Dozens of AI experts signed an open letter calling for research on the societal impacts of AI and concrete research directions.
The paper defined five practical safety problems (avoiding side effects, reward hacking, scalable oversight, safe exploration, robustness to distributional shift), setting the modern research agenda for the field.
Hendrycks et al. proposed a four-pillar taxonomy for the field: robustness, monitoring, alignment, and systemic safety.
During the AI Safety Summit the United States and the United Kingdom established their own AI Safety Institutes, giving the field an institutional and policy dimension.