Training
Next Token Prediction (NTP)
2018ActiveUpdated: 22 September 2026Published
Key
innovation
Turning simple next-token prediction into a scalable, universal training objective.
Category
Training
Abstraction level
Paradigm
Operation level
TrainingInference
Use cases
LLM pretrainingText generationAutoregressive modeling
How it works
The model predicts a distribution over the next token; the loss is cross-entropy against the actual next token in the data.
Problem solved
A scalable, self-supervised objective learnable from unlabeled text was needed.