Speech & Audio Processing

ASR

ActivePublished

Key innovation

Reducing speech-to-text to a single end-to-end trainable model (CTC, RNN-T, attention, Conformer), replacing the multi-stage HMM-GMM pipeline with its separate acoustic model, pronunciation lexicon and language model.

How it works

A typical ASR pipeline starts with a feature frontend: the signal is split into short frames and converted into log-mel filterbanks (formerly MFCCs). An acoustic encoder (RNN, Transformer or Conformer) turns the feature sequence into hidden representations. A training objective ties them to text via one of three approaches: CTC marginalises over all alignments using a blank token, RNN-T adds a prediction network and a joint network that emit output token by token, and attention-based encoder-decoder models decode autoregressively. Decoding uses beam search, often with shallow fusion of an external language model. Quality is evaluated with WER.

Problem solved

Speech is a time-varying, continuous signal with no clear word boundaries, and the same utterance varies with accent, speaking rate, noise and recording channel. ASR addresses the problem of mapping such a signal to discrete text without manual audio-to-letter alignment and in a way that is robust to this variability.

Components

Acoustic feature frontendFeature extraction

Converts the raw signal into feature frames (log-mel filterbanks / MFCCs).

Acoustic encoderAcoustic modelling

Models the feature sequence into hidden representations (RNN, Transformer, Conformer).

Alignment and decoding mechanismMapping to text

Maps acoustic representations to text tokens (CTC, RNN-T, attention).

Language modelLinguistic rescoring

Optional external language model improving linguistic correctness during decoding.

Official

Implementation

Reference implementations

C++ · Kaldi (community)

SpeechBrain

Python · SpeechBrain

Implementation pitfalls

Domain and acoustic mismatchHigh

WER rises sharply with noise, accents, overlapping speech and recording channels unlike the training data.

Fix:Data augmentation (noise, SpecAugment), fine-tuning on the target domain, speaker adaptation.

Transcription hallucinations and repetitionsMedium

Encoder-decoder models (e.g. Whisper) can generate text not grounded in the audio, especially on silence or long recordings.

Fix:Voice activity detection (VAD), repetition limiting, length penalties, timestamping.

Long-form audio and segmentationMedium

Models trained on short windows (e.g. 30 s in Whisper) require chunking long audio, which can cause boundary errors.

Fix:Overlapping windows, context-aware merging, boundary alignment.

Evolution

1952

Audrey system (Bell Labs)

Recognition of spoken digits from a single speaker — one of the first ASR systems.

1980

HMM-GMM paradigm

Inflection point

Statistical acoustic models based on hidden Markov models dominate ASR for decades.

2006

CTC (Connectionist Temporal Classification)

Inflection point

Sequence training without manual audio-to-label alignment — a foundation of end-to-end ASR.

Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks (paper)

2012

DNN-HMM and RNN-T

Inflection point

Deep neural acoustic models and RNN-T transducers (Graves) as an alternative to CTC.

Sequence Transduction with Recurrent Neural Networks (paper)

2014

Deep Speech

Large-scale end-to-end ASR based on CTC (Baidu).

Deep Speech: Scaling up end-to-end speech recognition (paper)

2015

Listen, Attend and Spell (LAS)

Attention-based encoder-decoder model for ASR.

Listen, Attend and Spell (paper)

2020

Conformer

Inflection point

Convolution-augmented Transformer; SOTA results on LibriSpeech.

Conformer: Convolution-augmented Transformer for Speech Recognition (paper)

2020

wav2vec 2.0

Inflection point

Self-supervised learning of speech representations reducing the need for labelled data.

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (paper)

2022

Whisper

Inflection point

Large-scale weakly-supervised training; robust, multilingual ASR (OpenAI).

Robust Speech Recognition via Large-Scale Weak Supervision (paper)

Hyperparameters (configurable axes)

Acoustic model architectureCritical

The network that encodes the acoustic signal.

Conformer

Transformer

RNN/LSTM

Training objective / decodingCritical

How acoustics are tied to text and output is produced.

CTC

RNN-T

attention encoder-decoder

Output unitsHigh

Output tokenization level.

znaki (characters)

podsłowa (BPE/WordPiece)

fonemy

Language model fusionMedium

Inclusion of an external language model during decoding.

brak

shallow fusion

rescoring

Feature frontendMedium

Input representation of the signal.

log-mel filterbanks (80-dim, 16 kHz)

MFCC

surowy przebieg (raw waveform)

Execution paradigm

Primary mode

Dense

Modern ASR models (Transformer/Conformer, CTC/RNN-T/attention) are dense neural networks; the whole network is active for each audio frame.

Activation pattern

All paths active

Parallelism

Parallelism level

Partially parallel

The acoustic encoder (especially Transformer/Conformer, CTC) processes frames in parallel during training, whereas autoregressive decoding (attention, RNN-T) is sequential token by token.

Scope

Training