Training
Online Distillation
2018ActiveUpdated: 15 September 2026Published
Key
innovation
Knowledge distillation carried out simultaneously with training, without a separate, fully pre-trained teacher.
Category
Training
Abstraction level
Pattern
Operation level
TrainingPost-training
Use cases
Efficient training of smaller modelsModel compressionEnsemble co-training of models
How it works
The student and teacher (or a group of peer models) train at the same time on the same data. During training the student matches its outputs to the teacher's soft predictions (or the ensemble's averaged predictions), transferring knowledge without a separate distillation phase.
Problem solved
Classic distillation requires a separate, fully trained teacher, which is costly and slow. Online distillation merges these stages.