Robots Atlas>ROBOTS ATLAS
Training

Self-improvement loop

2022ActivePublished: 6 August 2026Updated: 6 August 2026Published
Key innovation
Letting a model improve itself by learning from its own self-generated data (rationales, responses, judgements) - instead of relying solely on large human-labeled datasets.
Category
Training
Abstraction level
Pattern
Operation level
ModelDataEvaluation (runtime)
Use cases
Improving model reasoning (math, code, logic)Generating training data without human labelsTraining a successor model from a predecessor's dataPreference tuning without a fixed reward model (self-rewarding)Training agents on long-horizon tasks (e.g. whole repositories)

How it works

1) The model generates many candidate outputs for a task (e.g. different rationales/answers). 2) Outputs are scored or filtered: by verifiable correctness (e.g. a math answer, unit tests), an external verifier/reward model, or the model acting as a judge. 3) The best samples form a new training set. 4) The model is fine-tuned (e.g. supervised fine-tuning or DPO/RL) on that data. 5) The cycle repeats, and the improved model generates even better data in the next iteration. A reliable quality signal (verification) is crucial to avoid reinforcing errors.

Problem solved

Training strong models requires huge, high-quality human-labeled datasets that are costly and hard to scale - especially for advanced reasoning. A self-improvement loop lets training data be sourced from the model itself and scales quality without a proportional increase in human annotation.

Components

Generator (base model)Source of synthetic data.

The model producing candidate outputs (rationales, answers, solutions).

Verifier / judge / rewardThe quality signal that filters data.

A mechanism to score output quality: verifiable correctness, a reward model or LLM-as-a-Judge.

Official

Selection and dataset constructionData curation.

Selecting the best samples and building a new training set.

Official

Fine-tuning and iterationUpdating the model and closing the loop.

Fine-tuning the model (SFT/DPO/RL) on the selected data and repeating the loop.

Implementation

Implementation pitfalls
Error reinforcement / model collapseHigh

Training on a model's own unverified outputs can entrench errors and lead to quality degradation (model collapse).

Fix:A strong, independent verification signal (verifiable rewards, tests) and data filtering; mixing with external data.
Reward hacking / judge biasHigh

When a model judges itself, it may optimize for traits that fool the judge rather than real quality.

Fix:External/objective criteria, cross-verification, limiting the number of iterations.
Diversity narrowingMedium

Successive iterations can narrow the data distribution and reduce output diversity.

Fix:Sampling/temperature control, injecting fresh data, diversity monitoring.

Evolution

Original paper · 2022 · NeurIPS 2022 (arXiv:2203.14465) · Eric Zelikman
STaR: Bootstrapping Reasoning With Reasoning
Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman
2022
STaR - bootstrapping reasoning from a model's own data
Inflection point

Zelikman et al. showed a model can improve by learning from its own correct rationales - the foundation of the practical self-improvement loop.

2024
Self-Rewarding Language Models

The model judges its own outputs (LLM-as-a-Judge) and trains via DPO on self-generated rewards, improving iteratively without a fixed reward model.

2026
Self-improvement in production (Muse Spark 1.2)

Muse Spark 1.2 (Meta) uses self-improvement methods where prior models generate training data for later ones, co-trained with the Muse Code agent.