AV (Autonomous Vehicle)
How it works
The system runs a continuous real-time sense–plan–act loop. (1) Perception: sensors (cameras, LiDAR, radar, ultrasonics) provide raw data from which modules detect and track objects and build a semantic scene representation. (2) Localization and mapping: fusion of GNSS/IMU with HD maps and SLAM estimates the vehicle pose. (3) Sensor fusion: combines multi-modal readings into a coherent environment state. (4) Prediction: forecasts future trajectories of other road agents. (5) Planning: a behavioral layer selects a maneuver and a motion planner generates a safe, feasible trajectory. (6) Control: a controller (e.g. MPC) turns the trajectory into steering, throttle and brake commands. In the end-to-end approach much of this pipeline is replaced by a single neural network trained via behavior cloning or reinforcement learning.
Problem solved
Automating the driving task to reduce human error — the leading cause of road crashes — while increasing mobility access, optimizing traffic flow, and reducing labor costs in transport.
Components
Detects and tracks objects and builds a semantic scene representation from raw sensor data (cameras, LiDAR, radar, ultrasonics).
Official
Estimates the vehicle pose by fusing GNSS/IMU with high-definition (HD) maps and SLAM.
Official
Forecasts the future trajectories and intentions of other road agents (vehicles, pedestrians, cyclists).
Official
Selects a maneuver (behavioral layer) and generates a safe, feasible trajectory (motion planner).
Official
Turns the planned trajectory into actuator commands (steering, throttle, brake), often using MPC.
Official
Implementation
Rare, unusual traffic situations under-represented in training data cause perception and decision failures.
Rain, fog, snow, glare and dirt degrade the quality of camera and LiDAR data.
Behaviors learned in simulation do not transfer faithfully to the real world.
Statistically proving safety requires billions of kilometers of driving and coverage of rare events.
The whole perception→control pipeline must complete within tens of milliseconds at high speeds.
Evolution
Dean Pomerleau (CMU) trains a neural network mapping camera images to steering angle — a precursor of the end-to-end approach.
Ernst Dickmanns' team demonstrates high-speed vision-based driving in traffic (PROMETHEUS project).
Sebastian Thrun's Stanley completes the desert course; the first finished Grand Challenge.
CMU's Boss wins the urban race with traffic — a milestone for city driving.
Google launches its self-driving car program, spun out as Waymo in 2016.
SAE International defines a common taxonomy of six levels of driving automation.
A CNN maps raw camera pixels directly to steering commands — reviving the end-to-end approach.
Waymo launches a commercial autonomous ride-hailing service in the Phoenix area.
The "Planning-oriented Autonomous Driving" framework integrates the full driving-task stack in one network; presented at CVPR 2023.
Hyperparameters (configurable axes)
SAE J3016 automation level, from 0 (none) to 5 (full).
Range of conditions (area, weather, time of day, speeds) under which the system can operate safely.
Sensor modality configuration — from camera-only to camera + LiDAR + radar fusion.
Modular pipeline vs. end-to-end approach (a single neural network from sensors to control).
Degree of hardware and software redundancy ensuring safe operation after a fault.
Parallelism
The stack modules (perception, prediction, planning, control) run partly in parallel in real time, yet the control decision remains a sequential loop dependent on the latest perception.
Hardware requirements
Perception and planning networks run on automotive tensor-core accelerators (e.g. NVIDIA DRIVE, dedicated SoCs), delivering real-time inference within a constrained power budget.