The student and teacher (or a group of peer models) train at the same time on the same data. During training the student matches its outputs to the teacher's soft predictions (or the ensemble's averaged predictions), transferring knowledge without a separate distillation phase.
Classic distillation requires a separate, fully trained teacher, which is costly and slow. Online distillation merges these stages.
