1) The model generates many candidate outputs for a task (e.g. different rationales/answers). 2) Outputs are scored or filtered: by verifiable correctness (e.g. a math answer, unit tests), an external verifier/reward model, or the model acting as a judge. 3) The best samples form a new training set. 4) The model is fine-tuned (e.g. supervised fine-tuning or DPO/RL) on that data. 5) The cycle repeats, and the improved model generates even better data in the next iteration. A reliable quality signal (verification) is crucial to avoid reinforcing errors.
Training strong models requires huge, high-quality human-labeled datasets that are costly and hard to scale - especially for advanced reasoning. A self-improvement loop lets training data be sourced from the model itself and scales quality without a proportional increase in human annotation.
The model producing candidate outputs (rationales, answers, solutions).
A mechanism to score output quality: verifiable correctness, a reward model or LLM-as-a-Judge.
Official
Selecting the best samples and building a new training set.
Official
Fine-tuning the model (SFT/DPO/RL) on the selected data and repeating the loop.
Training on a model's own unverified outputs can entrench errors and lead to quality degradation (model collapse).
When a model judges itself, it may optimize for traits that fool the judge rather than real quality.
Successive iterations can narrow the data distribution and reduce output diversity.
Zelikman et al. showed a model can improve by learning from its own correct rationales - the foundation of the practical self-improvement loop.
The model judges its own outputs (LLM-as-a-Judge) and trains via DPO on self-generated rewards, improving iteratively without a fixed reward model.
Muse Spark 1.2 (Meta) uses self-improvement methods where prior models generate training data for later ones, co-trained with the Muse Code agent.
