A large model can teach a small one to nearly match it. Knowledge distillation trains a compact ‘student’ to mimic a big ‘teacher,’ capturing its behavior efficiently. It became a workhorse for deploying AI cheaply.
Teacher and student
Roles are clear. A big model’s outputs guide a small one. Behavior transfers.
Soft targets
The signal is rich. The teacher’s probability spread carries extra information. Students learn nuance.
Efficient deployment
The payoff is size. Small models run on modest hardware. Costs drop.
Minor accuracy loss
Trade-off is small. Students approach teacher performance. The gap is narrow.
Broad use
Adoption is wide. Vision and language models rely on it. Ubiquity followed.
Continued research
The idea evolves. New distillation methods appear. The field stays active.
The bottom line
Knowledge distillation trains a compact student to mimic a large teacher, capturing its behavior with minor accuracy loss. It enables cheap deployment. The technique is a widely used workhorse.
