E2E Learning
How it works
The entire processing pipeline is expressed as one differentiable model (or chain of layers). Raw data (pixels, audio samples, text) is fed as input, and the model produces the final task output. A single loss function is defined on the output; its gradient with respect to all model parameters is computed by backpropagation and the parameters are updated by gradient descent (e.g. SGD/Adam). Because the gradient flows through the whole model, all intermediate representations are learned jointly and tuned toward the final objective, with no hand-designed intermediate stages. The requirement is that the entire path from input to loss be differentiable.
Problem solved
Hand-engineered pipelines require expert feature engineering and separate optimization of each stage, which causes error accumulation and local optima misaligned with the end goal. End-to-end learning addresses this by optimizing the whole system against a single task-level loss, so intermediate representations are chosen automatically to serve the final outcome.
Components
One model spanning the whole pipeline, instead of a chain of separately engineered modules.
The model operates on raw data, without manual feature extraction.
One output loss replaces separate objectives for intermediate stages.
The loss gradient flows through the whole model, training all representations together.
Implementation
Training the whole pipeline from a single loss usually requires large labeled datasets.
Non-differentiable components break gradient flow and prevent end-to-end training.
The lack of explicit intermediate modules makes inspection and error diagnosis harder.
Evolution
LeCun et al. demonstrate training a full document recognition system with a single gradient.
CTC enables end-to-end training of sequence models without pre-segmented alignment.
Levine et al. train a robot policy from image directly to motor torques.
Amodei et al. demonstrate end-to-end speech recognition for English and Mandarin.
Bojarski et al. map camera pixels directly to steering commands.
Glasmachers analyzes the limits of the end-to-end paradigm versus modular approaches.