Sensor Fusion
How it works
1) Acquisition and preprocessing: each sensor provides measurements with its own noise model; data is filtered and prepared. 2) Synchronization and calibration: measurements are aligned in time (timestamping) and in space (mutual/extrinsic calibration) to a common reference frame. 3) State and observation models: a state vector is defined (e.g. position, velocity, orientation) together with a motion model (prediction) and an observation model mapping state to each sensor's measurements. 4) Recursive estimation: a Bayesian filter alternates a prediction step (propagating state and covariance via the motion model) and a correction step (updating the state with new measurements, weighted by their uncertainty — the Q and R covariances). The Kalman filter does this optimally for linear-Gaussian systems; the EKF linearizes the model, the UKF uses the unscented transform, and the particle filter represents the distribution with a set of samples for strongly nonlinear/non-Gaussian cases. 5) Data association: with multiple objects, measurements are assigned to the correct tracks. 6) Fusion level: combination can occur on raw data (early), on features, or on decisions (late), and architecturally in a centralized or distributed manner.
Problem solved
A single sensor is limited: it has finite accuracy, noise, range, field of view, and sensitivity to conditions (e.g. a camera degrades at night, LiDAR in rain/fog, GPS in tunnels). Fusion reduces uncertainty and noise, fills gaps in one source with information from others (complementarity), increases robustness to failures and interference through redundancy, and estimates quantities not directly measurable by any single sensor (e.g. position and velocity from combining IMU and GPS).
Components
The core of fusion — a Bayesian filter (Kalman, EKF, UKF, particle filter) alternately predicting and correcting the state estimate based on measurements.
A function mapping the state to the expected measurements of a given sensor, together with its noise model (covariance R).
A model predicting how the state evolves over time, with process noise (covariance Q), used in the prediction step.
Temporal (timestamping) and spatial (extrinsic/intrinsic calibration) alignment of measurements to a common reference frame.
Assigning measurements to the correct objects/tracks in multi-target tracking.
A description of estimate confidence (covariance matrices or probability distributions) that lets sources be weighted by their quality.
Implementation
Mismatched timestamps or latency between sensor streams cause measurements from different instants to be fused, degrading the estimate.
Inaccurate extrinsic/intrinsic sensor calibration causes systematic offsets when transforming into the common frame.
Incorrect process- and measurement-noise matrices make the estimate too sluggish or over-reactive, and in the extreme cause filter divergence.
Strong model nonlinearity makes the EKF's linearization produce biased estimates or divergence.
With multiple objects, incorrectly assigning measurements to tracks corrupts the entire tracking.
Evolution
Rudolf E. Kálmán introduces a recursive, optimal state estimator for linear-Gaussian systems — the mathematical foundation of modern sensor fusion.
The nonlinear filter variant (EKF) is used in Apollo navigation, popularizing fusion in spacecraft and aerospace guidance systems.
The Joint Directors of Laboratories (US DoD) formalize a multi-level data fusion model (levels 0-5), standardizing the field's terminology.
Gordon, Salmond, and Smith propose the particle filter for state estimation in strongly nonlinear, non-Gaussian systems.
Julier and Uhlmann introduce the unscented transform, giving more accurate nonlinear fusion without the explicit linearization required by the EKF.
The book Probabilistic Robotics (Thrun, Burgard, Fox) establishes Bayesian filters as the standard fusion approach in robot localization and mapping (SLAM).
Learned camera-LiDAR-radar fusion (including BEV-space representations and transformer models) becomes widespread in autonomous-vehicle perception.
Hyperparameters (configurable axes)
Abstraction level of the combined data: early (raw data), feature level, or late (decisions).
Estimator choice: Kalman (linear), EKF, UKF, or particle filter — depending on nonlinearity and noise distribution.
Encodes the motion-model uncertainty; tuning Q trades responsiveness against smoothness of the estimate.
Encodes each sensor's uncertainty; determines how strongly a given measurement corrects the state.
The selection and number of sources (camera, LiDAR, radar, IMU, GPS) determining complementarity and redundancy.
Centralized vs. distributed (decentralized) location where the data is combined.
Parallelism
Per-sensor preprocessing and distributed fusion can run in parallel (across nodes/devices), but the recursive filter correction step is inherently sequential in time.
Hardware requirements
Classical filters (Kalman/EKF/UKF) are computationally light and run on CPUs, microcontrollers, and embedded systems in real time.
The matrix operations of filters and state estimation benefit from SIMD vectorization on CPUs.
Neural-network-based fusion (e.g. BEV-space fusion, transformer models) benefits from GPU acceleration.