Robots Atlas>ROBOTS ATLAS
Training

OPSD

2026ResearchPublished: 2 September 2026Updated: 2 September 2026Published
Key innovation
A single model acts as both teacher and student: the teacher is conditioned on privileged information (e.g. verified reasoning traces or solutions), the student sees only the question, and distillation is performed over the student’s own on-policy trajectories — removing the need for a separate, larger teacher model.
Category
Training
Abstraction level
Pattern
Operation level
Post-trainingTraining
Use cases
LLM mathematical reasoningMulti-turn agentsReasoning-chain compression / concisenessMultilingual transferMultimodal LLMs

How it works

1) The student (model conditioned on the question only) samples its own trajectories — on-policy rollouts. 2) The same model, acting as teacher, is conditioned on privileged information (e.g. a verified reasoning trace or ground-truth answer). 3) For each token of the student’s trajectory, the divergence (typically reverse KL) between the student’s and the privileged teacher’s distributions is computed and minimized. 4) The teacher can be dynamically synchronized with the improving student, and the KL weight calibrated, to avoid training collapse.

Problem solved

Off-policy distillation trains the model on the teacher’s sequences, creating a train/inference distribution mismatch. On-policy distillation fixes this but needs a separate, usually larger teacher model and does not explicitly exploit the ground-truth solutions available in datasets. OPSD addresses both: teacher and student are the same model, and the knowledge comes from privileged information rather than from a bigger model.

Components

Student policyGenerates on-policy trajectories, conditioned only on the task prompt

Same network as the teacher but without access to privileged information; its distribution is the one being aligned.

Privileged teacherSame model conditioned on privileged information provides the per-token target

Sees verified reasoning traces / ground-truth solutions and produces the target distribution for each token.

Per-token distillation loss (reverse KL)Minimizes the divergence between student and teacher over the student’s trajectory tokens

Typically reverse KL; often importance-weighted for gradient correctness and stability.

Implementation

Evolution

Original paper · 2026 · arXiv (2026) · Siyan Zhao
Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, Aditya Grover
2023
GKD / on-policy distillation

Agarwal et al. introduce Generalized Knowledge Distillation: the student learns on its own sequences graded by a separate teacher.

On-Policy Distillation of Language Models (paper)
2025
On-policy distillation popularized

Thinking Machines Lab (Kevin Lu) describes on-policy distillation with dense per-token signal from a larger teacher.

On-Policy Distillation (Thinking Machines Lab) (paper)
2026
OPSD introduced
Inflection point

Named On-Policy Self-Distillation algorithm: a single model as teacher (privileged context) and student (question only).

Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (paper)
2026
OPSD extensions

Variants: privileged information distillation (π-Distill), Skill-SD for multi-turn agents, multimodal and multilingual variants.

Hyperparameters (configurable axes)

Privileged information typeCritical

What the teacher sees but the student does not: verified reasoning trace, ground-truth answer, extra context, higher resolution, etc.

KL divergence direction & weightHigh

Usually reverse KL between student and teacher; its weight/calibration drives stability and collapse risk.

Teacher–student synchronizationMedium

How often the teacher is updated to the student’s current state (static vs dynamic teacher).

Hardware requirements

Primary

LLM training/fine-tuning with a dense per-token loss requires tensor-core accelerators.