Knowledge Distillation: A Survey

Deep neural networks owe much of their success to scalability, encoding large-scale data with billions of parameters, but this same size creates high computational and storage demands that make deployment on resource-constrained devices such as mobile phones and embedded systems difficult. To address this, researchers have developed model compression and acceleration techniques. This survey focuses on knowledge distillation, a representative approach in which a compact student model is trained to learn from a larger, cumbersome teacher model.

The paper provides a comprehensive review of knowledge distillation organized around knowledge categories, training schemes, teacher-student architectures, distillation algorithms, performance comparisons, and applications. It also briefly reviews the challenges facing the field and offers commentary on promising directions for future research. As a broad, structured overview, it serves as a reference for understanding and comparing distillation methods.