BabyLM
How it works
The organizers release fixed training corpora and define tracks with a hard data cap. Participants pretrain any architecture using only that budget (with a limit of up to 10 epochs) and then submit the model to a shared evaluation pipeline. The main tracks are Strict (100M words, fixed dataset), Strict-Small (10M words, fixed dataset) and Loose — a track with the same limit on text quantity but freedom in the choice of data, its domain and even its modality (which in the 2024 edition evolved into a multimodal track: 100M words + paired image-text data). Models are ranked on the BLiMP, (Super)GLUE and EWoK benchmarks.
Problem solved
Modern language models require trillions of tokens and vast compute, making pretraining research inaccessible to most labs and detached from how humans learn language from limited input. BabyLM standardizes a comparable, small data budget and a shared evaluation, enabling rigorous study of data efficiency.
Components
Pretraining on a fixed, released corpus with a 100M-word budget.
Pretraining on a fixed, released corpus with a 10M-word budget — the most restrictive track.
Only the amount of text is capped; freedom in the choice of data, domain and modality. In 2024 it evolved into a multimodal track (100M words + paired image-text data).
A 100M-word text budget plus paired image-text data; evaluated on (visual) QA and grounding tasks among others. In 2024 no submission beat the baselines.
Shared evaluation pipeline: BLiMP (grammatical knowledge, minimal pairs), (Super)GLUE (language understanding after fine-tuning) and, since 2024, EWoK (Elements of World Knowledge).
Evolution
Inaugural challenge at CoNLL 2023 in Singapore. Strict (100M), Strict-Small (10M) and Loose tracks; evaluation on BLiMP and (Super)GLUE among others.
Second edition at CoNLL 2024 in Miami. Added a multimodal track (100M words + image) and the EWoK benchmark. Of 31 submissions, the best was the hybrid causal-masked model GPT-BERT.
The challenge continues as an annual workshop co-located with CoNLL/EMNLP, with evolving tracks and data budgets.
Hyperparameters (configurable axes)
Hard cap on training data: 10M or 100M words.
Choice of competition track: Strict, Strict-Small, Loose or Multimodal.
Training is limited to at most 10 epochs over the data budget.