1) The student (model conditioned on the question only) samples its own trajectories — on-policy rollouts. 2) The same model, acting as teacher, is conditioned on privileged information (e.g. a verified reasoning trace or ground-truth answer). 3) For each token of the student’s trajectory, the divergence (typically reverse KL) between the student’s and the privileged teacher’s distributions is computed and minimized. 4) The teacher can be dynamically synchronized with the improving student, and the KL weight calibrated, to avoid training collapse.
Off-policy distillation trains the model on the teacher’s sequences, creating a train/inference distribution mismatch. On-policy distillation fixes this but needs a separate, usually larger teacher model and does not explicitly exploit the ground-truth solutions available in datasets. OPSD addresses both: teacher and student are the same model, and the knowledge comes from privileged information rather than from a bigger model.
Same network as the teacher but without access to privileged information; its distribution is the one being aligned.
Sees verified reasoning traces / ground-truth solutions and produces the target distribution for each token.
Typically reverse KL; often importance-weighted for gradient correctness and stability.
Agarwal et al. introduce Generalized Knowledge Distillation: the student learns on its own sequences graded by a separate teacher.
Thinking Machines Lab (Kevin Lu) describes on-policy distillation with dense per-token signal from a larger teacher.
Named On-Policy Self-Distillation algorithm: a single model as teacher (privileged context) and student (question only).
Variants: privileged information distillation (π-Distill), Skill-SD for multi-turn agents, multimodal and multilingual variants.
What the teacher sees but the student does not: verified reasoning trace, ground-truth answer, extra context, higher resolution, etc.
Usually reverse KL between student and teacher; its weight/calibration drives stability and collapse risk.
How often the teacher is updated to the student’s current state (static vs dynamic teacher).
LLM training/fine-tuning with a dense per-token loss requires tensor-core accelerators.
