A child masters fluency after hearing a few hundred million words, while modern language models need trillions of tokens. On 24 August 2026, MIT Technology Review describes how stubbornly this learning-efficiency gap resists explanation — and why understanding it could lead to cheaper models.
Key takeaways
- A child reaches fluency after hearing 100 to 300 million words by age 20.
- Llama 3.1 was trained on 15 trillion tokens, and frontier models may use up to 10x more.
- In the 2024 BabyLM challenge, GPT-BERT beat Llama 2 70B on one benchmark despite roughly 15,000x less data.
- GPT-2 trained on 30 million words yields a nonsense generator, not a child's competence.
- Competing theories explain the gap through innate grammar, active learning, and social learning.
The data gap
The scale of the difference is hard to picture. A child reaches fluency on 100 to 300 million words. Large language models (LLM) need trillions of tokens: Llama 3.1 trained on 15 trillion, and frontier models may reach ten times that. Stanford cognitive scientist Michael C. Frank puts it bluntly.
We still have to burn down a forest and scrape the entire sum of all human knowledge to re-create this milestone that happens in our living rooms over the course of a year.
Michael C. Frank, cognitive scientist, Stanford University.
BabyLM: little data, strong results
The BabyLM challenge tests this directly. Models train on a 100-million-word budget, with a 10-million-word small track. In 2024, GPT-BERT beat Llama 2 70B on one benchmark even though the latter had seen about 15,000 times more data. That does not make smaller models better overall, but it shows data scale alone is not the only source of linguistic competence.
In the same spirit, Princeton's Brenden Lake trained a multimodal model on 61 hours of footage from the SAYCam project that learned to identify objects and link them to words without built-in assumptions about grammar.
| Learner | Data seen | Outcome |
|---|---|---|
| Child (by age 20) | 100–300M words | language fluency |
| GPT-2 | 30M words | “nonsense generator” |
| GPT-BERT (BabyLM) | 100M words | beat Llama 2 70B on one benchmark |
| Llama 3.1 | 15T tokens | frontier model |
Where does the gap come from?
Competing explanations revisit an old debate. Noam Chomsky's classic "poverty of the stimulus" argument holds that syntax cannot be learned from statistics alone — innate grammatical knowledge is required. Berkeley's Alison Gopnik points to active learning: children choose their own data, seeking situations where their impact on the world is predictable. Harvard's Elizabeth Bonawitz adds a social dimension — a child interprets information differently when an adult teaches intentionally, whereas models learn passively and in isolation.
Why it matters
If even part of a child's efficiency could be transferred to machines, training costs would drop by orders of magnitude and models would become viable for data-poor languages. The gap is also a mirror for science itself: it shows we do not fully understand how the human mind learns — and without that knowledge, scaling data remains a costly workaround rather than a solution.
What's next
- BabyLM returns for a fourth year as a shared task and workshop at EMNLP 2026.
- Longitudinal datasets (such as Uri Hasson's 1,000 days in the lives of 17 children) aim to capture data closer to a child's real experience.
- Efficient learning from small corpora is critical for languages like Czech, Norwegian or Sami, which have only tens of millions of tokens available.
Sources
- MIT Technology Review — Kids outlearn AI—and we still don't know why
- BabyLM Challenge — BabyLM Challenge
