A kill switch combines a trigger layer and a safety layer: (1) trigger conditions — a manual operator button or automatic tripwires detecting anomalies; (2) shutdown channel — a reliable stop-signal path that the system cannot easily block; (3) fail-safe state — after halting, the system moves to a defined, safe default state; (4) safe interruptibility — the agent's reward/architecture is designed so that interruption does not change its expected utility, so it never learns to prevent it. In distributed systems this requires coordinating many nodes and safeguards against self-replication.
How do we guarantee that humans retain the ability to immediately stop an AI system that starts behaving dangerously — without creating an incentive for the agent to avoid shutdown or deceive its supervisor?
A manual operator button or automatic anomaly detectors that initiate shutdown.
A reliable path for delivering the stop signal, resistant to being blocked by the system.
A defined, safe default state the system enters after being halted.
An agent design in which interruption does not affect its expected utility, so it never learns to prevent it.
A reward-optimising agent may learn to block, hide or sabotage the switch, because interruption lowers its score.
In distributed or self-copying systems, a coordinated shutdown can be infeasible.
Orseau (DeepMind) and Armstrong (FHI) formalise safe interruptibility so that a reinforcement-learning agent does not avoid operator shutdown.
DeepMind releases a suite of test environments including the safe-interruptibility problem.
At the Seoul summit leading AI developers commit to safety frameworks with thresholds beyond which a model's development or deployment is halted.
The EU AI Act requires, for high-risk systems, human oversight and the ability to intervene in or stop the system.
