Safety-Critical Autonomous Systems (SCAS)
How it works
SCAS architecture is layered. (1) Perception layer: redundant, diverse sensors (RGB cameras, LiDAR, radar, IMU) โ no single modality is a critical point of failure. (2) AI/ML layer: the main decision stack (planner, perception, prediction) uses deep models, but its outputs are always observed. (3) Runtime monitoring layer: OOD detectors, uncertainty monitors (epistemic + aleatoric), statistical model checking, safety-predicate evaluation (e.g. "speed < limit", "distance > minimum"). (4) Safety cage / simplex layer: a deterministic backup controller that can take over if AI behaves anomalously or enters an undefined state โ e.g. stop the vehicle, return manual control to the pilot, alert the operator. (5) Validation: combination of classical methods (HIL testing, millions of km/hours of field testing) with new AI-specific techniques (adversarial testing, scenario-based testing, fuzz testing, formal verification of critical components). (6) Certification: documenting the evidence chain in line with industry standards and new ones (ISO 21448 SOTIF, ISO 8800, EU AI Act Annex III).
Problem solved
Classical functional-safety methods do not directly apply to AI systems: one cannot prove correctness of neural-network weights, there are no deterministic error bounds, and behavior on out-of-distribution inputs is unpredictable. SCAS addresses the question: how to introduce AI into systems whose failure can kill, while preserving the level of rigor required by standards and regulators.
Components
A deterministic supervisor controller that observes AI outputs and can take over when behavior leaves the safety envelope. The simplex architecture (Sha 2001) โ a simple, provably correct fallback controller alongside a complex but non-deterministic AI controller.
Continuous evaluation of safety predicates, epistemic uncertainty, input-distribution drift (OOD detection), and consistency of outputs with the nominal model. Generates escalation signals to the supervisor layer.
Official
Multiple independent decision stacks (e.g. two planning systems built on different models), diverse sensors, and independent actuation paths. Eliminates single points of failure (SPOF).
Official
Provable verification of critical safety properties (e.g. "the car never enters an intersection on red") using model checking, SMT solvers, or theorem provers. Applied to deterministic layers, not the network itself.
Official
Compliance with industry standards: ISO 26262 (ASIL AโD), ISO 21448 (SOTIF โ safety of the intended functionality), ISO 8800 (AI in automotive), DO-178C / DO-254 (aviation), IEC 61508 (SIL 1โ4 industrial), IEC 62304 (medical). Plus EU AI Act for high-risk systems.
Implementation
The production environment contains scenarios absent from training (rare weather, unusual road situations, mud-covered sensors). AI loses confidence or produces wrong predictions.
Two "independent" decision stacks can fail simultaneously due to the same cause โ e.g. both use the same library, training data, or backbone network.
The most dangerous scenarios (accidents, extreme conditions) are inherently rare. The dataset has strong imbalance โ the model mostly learns "nominal" scenarios.
Runtime monitor detects the problem too late; the simplex layer cannot stop the system before consequences materialise.
Evolution
Therac-25 radiation therapy failures killed patients due to software errors, exposing the inadequacy of classical software engineering in safety-critical systems and forcing the field to develop.
First standard defining SIL (Safety Integrity Level) tiers and a framework for electrical/electronic/programmable systems. Became the template for industry-derived standards.
Lui Sha proposes the formal safety-cage architecture โ a simple, provably correct controller alongside a complex non-deterministic one. The architectural foundation of modern SCAS.
Launch of the automotive standard with ASIL AโD levels. Became the industry standard for every carmaker and gave rise to a whole ecosystem of tools and processes.
Pedestrian Elaine Herzberg died after being struck by an Uber autonomous car. Investigation revealed the AI system "dismissed" a real object as a false positive and the human operator was not ready to take over. Inflection point in the discussion about SCAS for cars.
Standard complementing ISO 26262 with "Safety Of The Intended Functionality" โ addresses the safety of correctly functioning features under uncertainty (e.g. AI perception in unusual weather).
Introduction of the EU AI Act (high-risk AI systems regulation) and ISO 8800 specific to AI in automotive. First serious regulatory and normative framework for AI/ML-based SCAS.
Waymo scaled commercial robotaxi service in several US cities, and humanoids (Figure 02, Apptronik Apollo) started working in BMW and Mercedes factories โ the first cases of largely-neural-network SCAS in commercial services.
Hyperparameters (configurable axes)
Required level of rigor per industry standard. Affects development process, redundancy, audit frequency, and allowed failure rate (1e-9 / hour for DAL A in aviation).
What the system does when the safety monitor detects an anomaly. The specific mode depends on the domain โ a car can pull over, an aircraft can hand control back to the pilot, a robot can freeze.
Number of independent decision paths. Higher redundancy increases MTBF and detects systematic errors but raises cost and complexity.
Execution paradigm
The system switches between nominal mode (AI in charge), cautious mode (AI under strict monitoring), and emergency mode (deterministic fallback controller) depending on safety-monitor state.
Parallelism
The main AI stack and runtime monitoring can run in parallel on different compute units. The simplex layer is a sequential supervisor; its overriding decision cannot be skipped due to latency.
Hardware requirements
The main AI stack (perception, prediction, planning) runs on GPUs. Often automotive-certified GPUs are used (NVIDIA DRIVE).
The simplex / safety-cage / runtime-monitor layer typically runs on lockstep CPUs with hard real-time, for full provability.
FPGAs are used for deterministic, verifiable sensor signal processing (LiDAR, radar) and as a platform for simplex.