A VSR pipeline starts with face detection and localization of a region of interest (ROI) around the mouth, followed by alignment and normalization of the frame sequence. A visual front-end (spatiotemporal convolution / 3D-CNN) extracts lip-motion features from the image sequence. A sequence model (GRU/LSTM or Transformer/Conformer) models the temporal dependencies between frames. A decoder maps the feature sequence to text, using a CTC loss (which allows differing input and output lengths) or a sequence-to-sequence architecture with attention. Because many phonemes produce the same lip shape (homophenes/visemes), the model must exploit language context. More recent approaches (e.g. AV-HuBERT) use self-supervised pretraining on large audio-visual corpora to alleviate the scarcity of labeled data.
Speech recognition fails or is unavailable when the audio signal is noisy, masked by the environment, or entirely absent (e.g. silent video, a noisy hall, several people speaking at once). VSR recovers the spoken content from the image of the lips alone and can indicate who is currently speaking, improving the robustness of speech-recognition and human–robot interaction systems in difficult acoustic conditions.
A convolutional network operating over spatial and temporal dimensions, turning the cropped mouth region into a sequence of feature representations.
Official
A recurrent or attention-based layer that aggregates features over time into a contextual sequence representation.
Official
Decoding module: a CTC loss allowing input/output length mismatch, or a seq2seq decoder with attention generating characters/words.
Official
Many distinct phonemes produce the same visible lip shape, making decoding ambiguous.
Poor face/landmark detection or unstable cropping degrades the input.
Variation in recording conditions and speaker traits harms generalization.
Annotating speech video is costly, limiting supervised training.
Release of the GRID audio-visual corpus (Cooke et al.), a standard word/sentence-level VSR benchmark.
First end-to-end sentence-level VSR (spatiotemporal CNN + GRU + CTC), reaching 95.2% accuracy on GRID.
Deep Audio-Visual Speech Recognition (Afouras et al.) — in-the-wild VSR using Transformers, plus the LRS2-BBC and LRS3-TED datasets.
Self-supervised learning via masked multimodal cluster prediction; 26.9% WER on LRS3 (433 h).
How the mouth region is detected, cropped and aligned — strongly affects quality.
Choice of front-end and sequence model (3D-CNN, Transformer, Conformer).
CTC vs sequence-to-sequence with attention.
Granularity of the recognized output units.
Transformer/Conformer variants train in parallel across frames; recurrent variants are sequential over time.
Training and inference of spatiotemporal CNNs and Transformers over video sequences.
