Instead of enlarging the training set, researchers work on extracting more learning signal from each example. Common levers are: (1) richer training objectives — e.g. a hybrid of masked (BERT) and causal (GPT) language modeling, as in GPT-BERT; (2) curriculum learning — ordering data from easier to harder; (3) data augmentation and careful preprocessing; (4) architectures optimized for small corpora (e.g. LTG-BERT); (5) regularization and multiple passes over the data without overfitting; (6) incorporating multimodal signal. Models are evaluated on standardized benchmarks (e.g. BLiMP, (Super)GLUE, EWoK in the BabyLM pipeline) to compare quality under an equal, constrained data budget.
Modern LLMs achieve high quality mainly by scaling data to hundreds of billions or trillions of tokens, which is expensive, inaccessible to most teams, and implausible as a model of human language acquisition. Sample-Efficient Pretraining addresses how to reach good quality with data on the scale of a child's language exposure (10–100M words).
Richer loss functions that extract more signal per example, e.g. combining masked (MLM) and causal (CLM) language modeling.
Ordering training data from easier to harder, inspired by a child's exposure order.
Creating additional data variants and careful preprocessing to increase effective coverage under a fixed word budget.
Architectures optimized for small corpora, e.g. LTG-BERT or the hybrid GPT-BERT.
Stronger regularization and multiple passes over the same set without overfitting, to fully exploit a small corpus.
Using signal from other modalities (e.g. vision) as additional context when text is limited.
With 10–100M words, large models easily overfit the data.
Without standardized benchmarks it is hard to fairly compare quality at an equal data budget.
Inadvertently introducing extra data (e.g. via pretrained components) invalidates the comparison.
Gains from curriculum learning can be unstable and task-dependent.
Defined strict data tracks: Strict-Small (10M words) and Strict (100M words); established the sample-efficient pretraining paradigm at CoNLL.
A hybrid of masked and causal language modeling outperforms MLM-only and CLM-only variants under the same data budget.
The challenge continues as a workshop, maintaining the focus on sample-efficient pretraining on developmentally plausible corpora.
Size of the training corpus, e.g. 10M words (Strict-Small) or 100M words (Strict).
Ratio between masked and causal language modeling (e.g. in GPT-BERT).
Parameter count chosen so as not to overfit the small corpus.
How data is ordered from easy to hard.
How many times the model passes over the same small dataset.
The paradigm inherits transformer training parallelism; the bottleneck is the data budget, not compute parallelism.
The small data budget enables training on a modest single GPU in academic settings.
Training the underlying transformer still benefits from GPU accelerators.
