Gloss
How it works
An annotator writes each manual sign as a spoken-language word in ALL CAPS in signing order; non-manual features and prosody are noted as superscripts with scope in brackets, fingerspelling with hyphens, and lexicalized forms with #. In the ML pipeline: (1) sign2gloss2text — a visual network recognizes a gloss sequence from video (usually trained with a CTC loss since there is no frame-level alignment) and a second network translates the glosses into a sentence; (2) in joint models (Sign Language Transformers) the gloss acts as an auxiliary CTC supervision target alongside the main translation objective. Because gloss order differs from spoken word order (non-monotonicity), the gloss-to-text stage performs full sequence-to-sequence translation.
Problem solved
Sign languages have no widely adopted written form, and continuous signing is hard to align to spoken text. A gloss provides a discrete, tokenized mid-level representation that enables supervised recognition and translation and quantitative evaluation of systems.
Components
A manual sign written as a capitalized spoken-language word, placed in signing order.
Usually written as superscripts with scope in brackets, e.g. [I LIKE]^negative.
Pure fingerspelling marked with hyphens (W-I-K-I); lexicalized forms marked with a hash (#JOB).
Partial encoding of spatial grammar; inherently incomplete, which is a source of information loss.
Implementation
Gloss flattens simultaneity, spatial grammar and non-manual features, creating an information bottleneck that limits translation quality.
Gloss annotation requires trained linguists or native signers; datasets remain small (on the order of 10–20k pairs).
Conventions differ across corpora and sign languages, hurting reproducibility and cross-dataset transfer.
Gloss order differs from spoken word order, so gloss-to-text alignment is non-trivial.
Evolution
German TV weather forecasts annotated with glosses — a standard benchmark for continuous sign language recognition.
Camgöz et al. formalize gloss as a mid-level representation in neural sign language translation.
Joint end-to-end recognition and translation; gloss as an auxiliary CTC target drastically improves translation quality.
Chen et al. connect a sign-to-gloss and a gloss-to-text network with a visual-language mapper, boosting the gloss pipeline via transfer learning.
Zhou et al. point to the "information bottleneck" of gloss representation and data scarcity, proposing gloss-free translation.
Extends FLORES/FLEURS with American Sign Language as video; shows frontier models have virtually no understanding of ASL.
Hyperparameters (configurable axes)
No universal standard — conventions differ across corpora and sign languages (ASL, DGS, BSL).
The number of unique glosses in a corpus affects recognition difficulty and coverage.
Whether and how finely facial expression, prosody and spatial grammar are noted alongside manual glosses.