Safety Case
How it works
A safety case is typically built in a manner similar to Goal Structuring Notation: it starts from a top-level claim (e.g. "deploying this system will not cause a catastrophe in this context") that is decomposed into progressively more specific sub-claims. Each claim is linked to evidence through an explicit argument (an inference rule), and the evidence comes from capability evaluations, red-team testing, interpretability research, and measurements of safeguard and control effectiveness. In Clymer et al.'s framing, arguments fall into four categories: (1) inability โ the system cannot cause a catastrophe; (2) control โ even if it has the capability, control measures prevent its use; (3) trustworthiness โ the system would not use its capability to cause harm because it is trustworthy; (4) deference โ relying on the judgement of credible AI advisors. Arguments can be combined; the case explicitly states its assumptions and boundary conditions (scope of application, operating environment), and must be revisited whenever the system or its context of use changes.
Problem solved
Individual evaluations and benchmarks reveal isolated capabilities or behaviours, but on their own they do not prove that the system as a whole will not cause serious harm once deployed. A safety case ties scattered evidence into a single explicit, falsifiable argument, surfaces its underlying assumptions and gaps, and gives regulators and decision-makers a transparent basis for deciding whether to train or deploy a system.
Components
A top-level safety claim decomposed into a hierarchy of sub-claims that together are meant to justify an acceptable level of risk.
Explicit inference rules linking evidence to claims. In Clymer et al.'s framing they fall into four categories of safety justification.
The empirical basis of the argument: capability evaluation results, red-team tests, interpretability research, and measurements of safeguard and control effectiveness.
Evolution
Safety cases operate as a required safety argument in industries such as aviation, nuclear power, rail and offshore (e.g. the UK Offshore Installations (Safety Case) Regulations 1992).
Clymer, Gabrieli, Krueger and Larsen propose a framework for organizing an AI safety case and four categories of argument: inability, control, trustworthiness, deference.
Balesni, Hobbhahn et al. (Apollo Research) propose how to construct evaluations-based safety cases for the risk of AI scheming.
The UK AI Safety Institute (Goemans, Buhl, Irving et al.) publishes a concrete frontier-AI safety case template built on a cyber inability argument.