1. Train the teacher: a large model (e.g. 70B) is trained standardly on a large dataset to high quality. 2. Prepare the distillation dataset: for each training example, the teacher produces soft targets - a probability distribution over classes (softmax(logits/T) where T > 1 raises entropy). 3. Train the student: a smaller model (e.g. 7B) is trained on a combination of two losses: (a) L_KD = KL(soft_teacher || soft_student) - match the teacher's distribution; (b) L_CE = cross_entropy(hard_labels, student) - match the true labels. Total loss: α * L_KD + (1-α) * L_CE, typically α ≈ 0.7. 4. Extended variants: distillation of intermediate features (hidden states), attention maps, RLHF-style with a reward model, on-policy distillation (the student generates rollouts, the teacher scores them). 5. LLM context 2024-2026: dataset distillation - the teacher generates synthetic (prompt, response) pairs, and the student is fine-tuned on them (Alpaca, Zephyr, DistilABC approaches).
Large AI models (billions of parameters) are too expensive for edge/mobile inference and too costly to serve at massive scale. KD transfers their capabilities to models 5-100x smaller while retaining 90-99% of the quality - critical for production, on-device deployment, and inference cost.
A large, well-trained model (e.g. GPT-4, Claude Opus 4.7, Llama-70B) whose capabilities we want to transfer.
Official
A smaller model (5-100x fewer parameters), typically of the same or similar architecture as the teacher; trained from scratch or from initialisation.
Official
Combination of KL-divergence between the teacher's soft targets and the student's output plus cross-entropy to hard labels. Key hyperparameter: α (weight of KD vs CE).
Hyperparameter T > 1 that smooths the softmax: softmax(logits/T). Higher T = more uniform distribution = more dark knowledge but a less crisp signal.
A set of (input, teacher_soft_output, hard_label) triples - can be the original training dataset, unlabelled data with inferences, or synthetic data generated by the teacher (dataset distillation).
Official
When the teacher is too large relative to the student (>100x), the student cannot 'fit' the knowledge - KD quality drops sharply.
Too low temperature = classic CE (no dark knowledge). Too high = loss of preference signal - distribution near uniform.
Distillation across different architectures (Transformer → Mamba) requires translating intermediate representations - naive feature distillation will not work.
A teacher trained on one data distribution may give uncertain soft targets on out-of-distribution examples - transfer fails.
The student inherits all of the teacher's errors, hallucinations, and biases - including ones absent from the original training data.
Bucila, Caruana, Niculescu-Mizil publish 'Model Compression' - a predecessor to distillation: a small model learns to imitate an ensemble of larger models.
Hinton, Vinyals, Dean formalise KD with temperature-softened softmax and introduce the term 'dark knowledge'. Foundation for all subsequent work.
Sanh et al. (Hugging Face) release DistilBERT - 40% smaller, 60% faster than BERT with 97% of the results. First mass-used KD application in NLP.
Extensions of distillation to intermediate layers (TinyBERT) and attention maps (MobileBERT). Establishes 'feature distillation' as a sub-class.
Paradigm shift in LLMs: instead of soft targets, the teacher (GPT-4) generates synthetic (prompt, response) pairs; the student is fine-tuned classically. Opens 'behaviour distillation' without access to teacher weights.
Google releases Gemma trained via distillation from Gemini, Microsoft releases Phi-3 distilled from GPT-4. Establishment: distillation is a standard part of training small flagship models.
DeepSeek releases DeepSeek-R1-Distill (Qwen/Llama distilled from R1). Thinking Machines Lab introduces on-policy distillation - the student generates rollouts, the teacher scores them, used in post-training of reasoning models.
Distribution Matching Distillation 2 - evolution of KD for generative models (image/video diffusion). Another inflection point combining diffusion and KD.
Time complexity: O(T_student · N_examples) + koszt inferencji nauczyciela. Space complexity: O(|θ_student|) w produkcji; O(|θ_teacher| + |θ_student|) w trakcie treningu.
Logit scaling before softmax. T > 1 smooths the distribution; T = 1 is the standard softmax.
Loss weight: total_loss = α * L_KD + (1-α) * L_CE. Higher α = more learning from the teacher.
How many times larger the teacher is than the student. Too large a gap (>100x) can yield poor distillation - the capacity gap problem.
KD is neutral to the student's architecture - it can be dense (classic DistilBERT), sparse (distillation to MoE), and even architecture conversion (Transformer → Mamba).
Soft targets can be generated offline via batch teacher inference on separate GPUs; the student's training is then fully parallel like any training.
Training the student and running teacher inference are standard matmul operations - Tensor Cores are optimal.
Google (creator of Gemma) uses TPUs for all distillation training - excellent for batch generation of soft targets.
KD as a training paradigm has no hardware preference - what matters is teacher+student size in memory.
