Robots Atlas>ROBOTS ATLAS
Evaluation

WER

ActivePublished
Key innovation
Reducing speech transcription quality to a single, reproducible number based on word-level edit distance, accounting for substitutions, deletions and insertions.
Category
Evaluation
Abstraction level
Benchmark
Operation level
Evaluation (runtime)Inference
Use cases
Evaluating ASR (speech-to-text) systemsComparing speech recognition models (e.g. Whisper, NVIDIA Parakeet)Monitoring transcription-quality regressionsASR benchmarks (e.g. LibriSpeech)Assessing quality across languages and acoustic conditions

How it works

1) The hypothesis transcript (ASR output) and the reference transcript are normalized (e.g. lowercasing, punctuation removal). 2) The two word sequences are aligned by computing the minimum word-level edit (Levenshtein) distance. 3) Three error types are counted: substitutions (S — a word replaced by another), deletions (D — a reference word omitted) and insertions (I — a word added by the system). 4) The error sum is divided by the number of reference words N: WER = (S + D + I) / N. The metric is based on Levenshtein distance and is normally reported after the same normalization across all compared systems.

Problem solved

An objective, reproducible and cross-model way of measuring speech transcription accuracy is needed. Subjective 'by ear' judgement cannot reliably compare systems or track progress. WER reduces ASR quality to a single number that accounts for three error types — substitutions, deletions and insertions — against a reference transcript.