Self-Distillation
How it works
Instead of a separate teacher, the supervisory signal comes from the model itself. (1) Iterative variant (Born-Again): a model is trained, then an identical copy is trained using the softened predictions of the previous generation as targets; the process can be repeated. (2) In-network variant (Be Your Own Teacher): the network is split into sections with auxiliary classifiers; deeper (more accurate) sections act as a teacher for shallower ones via a distillation loss on logits and/or features. (3) Prediction-based variant: the model learns from its own averaged/softened outputs. In all variants the loss combines the ordinary hard-label loss with a KL divergence between student and teacher predictions, scaled by a softmax temperature.
Problem solved
Classic knowledge distillation requires training a separate, large teacher model, which is costly and not always feasible. Self-distillation removes that dependency: it improves a model’s accuracy and regularization without an extra network, and the in-network variant enables depth-wise scalable inference on edge devices.
Components
The source of soft targets coming from the model itself: deeper layers, a previous generation of weights, or averaged/softened predictions.
The same or identically parameterized model (or its shallower section) trained to match the self-teacher’s soft targets.
A combination of a hard-label loss (cross-entropy) and a KL divergence between the softened predictions of student and teacher, scaled by temperature.
Implementation
A temperature that is too low or too high degrades the soft-target signal.
Overweighting the KL loss can hurt hard-label accuracy.
Successive generations (Born-Again) multiply training time.
Evolution
Introduction of knowledge distillation with a separate teacher and soft targets — the base mechanism for self-distillation.
A student identical to the teacher outperforms it — an early, iterative variant of self-distillation.
In-network self-distillation: deeper layers teach shallower ones; coined the “self distillation” method.
Theoretical explanation: self-distillation iterations progressively limit the function basis, amplifying regularization.
Hyperparameters (configurable axes)
Scales the softness of teacher predictions; critical for the quality of the distillation signal.
Balance between the hard-label loss and the distillation (KL) loss.
Number of self-distillation iterations in the Born-Again variant.
Splitting the network into sections with auxiliary classifiers in the in-network variant.
Parallelism
The in-network (deep→shallow) variant trains in parallel in a single run; the iterative (Born-Again) variant is sequential across generations.
Hardware requirements
A training technique independent of hardware; runs on any accelerator (most commonly GPU).