KD
How it works
1. Train the teacher: a large model (e.g. 70B) is trained standardly on a large dataset to high quality. 2. Prepare the distillation dataset: for each training example, the teacher produces soft targets - a probability distribution over classes (softmax(logits/T) where T > 1 raises entropy). 3. Train the student: a smaller model (e.g. 7B) is trained on a combination of two losses: (a) L_KD = KL(soft_teacher || soft_student) - match the teacher's distribution; (b) L_CE = cross_entropy(hard_labels, student) - match the true labels. Total loss: α * L_KD + (1-α) * L_CE, typically α ≈ 0.7. 4. Extended variants: distillation of intermediate features (hidden states), attention maps, RLHF-style with a reward model, on-policy distillation (the student generates rollouts, the teacher scores them). 5. LLM context 2024-2026: dataset distillation - the teacher generates synthetic (prompt, response) pairs, and the student is fine-tuned on them (Alpaca, Zephyr, DistilABC approaches).
Problem solved
Large AI models (billions of parameters) are too expensive for edge/mobile inference and too costly to serve at massive scale. KD transfers their capabilities to models 5-100x smaller while retaining 90-99% of the quality - critical for production, on-device deployment, and inference cost.
Components
A large, well-trained model (e.g. GPT-4, Claude Opus 4.7, Llama-70B) whose capabilities we want to transfer.
Official
A smaller model (5-100x fewer parameters), typically of the same or similar architecture as the teacher; trained from scratch or from initialisation.
Official
Combination of KL-divergence between the teacher's soft targets and the student's output plus cross-entropy to hard labels. Key hyperparameter: α (weight of KD vs CE).
Hyperparameter T > 1 that smooths the softmax: softmax(logits/T). Higher T = more uniform distribution = more dark knowledge but a less crisp signal.
A set of (input, teacher_soft_output, hard_label) triples - can be the original training dataset, unlabelled data with inferences, or synthetic data generated by the teacher (dataset distillation).
Official
Implementation
When the teacher is too large relative to the student (>100x), the student cannot 'fit' the knowledge - KD quality drops sharply.
Too low temperature = classic CE (no dark knowledge). Too high = loss of preference signal - distribution near uniform.
Distillation across different architectures (Transformer → Mamba) requires translating intermediate representations - naive feature distillation will not work.
A teacher trained on one data distribution may give uncertain soft targets on out-of-distribution examples - transfer fails.
The student inherits all of the teacher's errors, hallucinations, and biases - including ones absent from the original training data.
Evolution
Bucila, Caruana, Niculescu-Mizil publish 'Model Compression' - a predecessor to distillation: a small model learns to imitate an ensemble of larger models.
Hinton, Vinyals, Dean formalise KD with temperature-softened softmax and introduce the term 'dark knowledge'. Foundation for all subsequent work.
Sanh et al. (Hugging Face) release DistilBERT - 40% smaller, 60% faster than BERT with 97% of the results. First mass-used KD application in NLP.
Extensions of distillation to intermediate layers (TinyBERT) and attention maps (MobileBERT). Establishes 'feature distillation' as a sub-class.
Paradigm shift in LLMs: instead of soft targets, the teacher (GPT-4) generates synthetic (prompt, response) pairs; the student is fine-tuned classically. Opens 'behaviour distillation' without access to teacher weights.
Google releases Gemma trained via distillation from Gemini, Microsoft releases Phi-3 distilled from GPT-4. Establishment: distillation is a standard part of training small flagship models.
DeepSeek releases DeepSeek-R1-Distill (Qwen/Llama distilled from R1). Thinking Machines Lab introduces on-policy distillation - the student generates rollouts, the teacher scores them, used in post-training of reasoning models.
Distribution Matching Distillation 2 - evolution of KD for generative models (image/video diffusion). Another inflection point combining diffusion and KD.
Hyperparameters (configurable axes)
Logit scaling before softmax. T > 1 smooths the distribution; T = 1 is the standard softmax.
Loss weight: total_loss = α * L_KD + (1-α) * L_CE. Higher α = more learning from the teacher.
How many times larger the teacher is than the student. Too large a gap (>100x) can yield poor distillation - the capacity gap problem.
Computational complexity
Time complexity: O(T_student · N_examples) + koszt inferencji nauczyciela. Space complexity: O(|θ_student|) w produkcji; O(|θ_teacher| + |θ_student|) w trakcie treningu.
Compute bottleneck
The main cost of distillation is running forward passes of the large teacher model over the training data to obtain soft targets (soft labels / logits). Teacher inference dominates the compute and memory of the process, especially with large teachers and large datasets.
Execution paradigm
KD is neutral to the student's architecture - it can be dense (classic DistilBERT), sparse (distillation to MoE), and even architecture conversion (Transformer → Mamba).
Parallelism
Soft targets can be generated offline via batch teacher inference on separate GPUs; the student's training is then fully parallel like any training.
Hardware requirements
Training the student and running teacher inference are standard matmul operations - Tensor Cores are optimal.
Google (creator of Gemma) uses TPUs for all distillation training - excellent for batch generation of soft targets.
KD as a training paradigm has no hardware preference - what matters is teacher+student size in memory.