A large model’s competence need not stay large. Its behavior can be poured into a smaller vessel that mimics it closely, keeping most of the skill at a fraction of the size. Knowledge, it turns out, is portable.
The claim
Distillation trains a compact model to imitate a larger one rather than to learn from raw labels alone. The small model watches the large one’s outputs and matches them. It inherits behavior without inheriting bulk.
Beneath the surface
The mechanism is richer supervision. The large model’s graded, probabilistic outputs carry more information than a bare correct answer. The student learns not only what is right but how confident and how close the alternatives are.
A reframing
The reframing is that learning targets can themselves be learned. The best teacher for a small model may be a large one, not the original data. Understanding becomes something transferable between artifacts.
The trade-off
Distillation is imperfect. The student rarely matches the teacher exactly, and some capability is lost in the transfer. The compression is lossy, and the loss is not always where one expects.
Where it breaks
The implication is that size and skill are separable. A model can be large to learn and small to serve, decoupling the cost of acquiring competence from the cost of using it. Deployment need not carry the weight of training.
The larger point
Distillation transfers a large model’s behavior into a compact one through richer, graded supervision. It separates the cost of learning from the cost of serving. Competence proves surprisingly portable, if imperfectly so. What makes the idea durable is not that it settles a question but that it reframes many. It teaches where to look and what to discount, which is often more valuable than any particular answer it yields. Understood in this spirit, it becomes a habit of attention rather than a doctrine, and habits of attention are what distinguish deep comprehension from mere knowledge.