Adding prediction of a 'concept' (a multi-token latent representation) alongside token prediction.
Category
Architecture
Abstraction level
Paradigm
Operation level
TrainingModel
Use cases
Latent language modelsReasoning and planningLLM pretraining
How it works
The model learns to predict the next quantized concept (a latent representation of several tokens), which steers the decoder that generates subsequent tokens.
Problem solved
Predicting only the next token limits longer-horizon planning and abstract reasoning.
