distilbert

DistilBERT is a smaller, faster, and lighter version of BERT produced through knowledge distillation — a model compression technique where a smaller "student" model is trained to replicate the behavior of a larger "teacher" model. Created by Hugging Face and introduced by Sanh et al. (2019), DistilBERT retains 97% of BERT's language understanding capability while being 60% smaller and 60% faster, making it practical for deployment in resource-constrained environments. The distillation process involves training the student model on three combined objectives: distillation loss (soft target probabilities — the student learns to match the teacher's output probability distribution, which contains richer information than hard labels because it captures relationships between classes), masked language modeling loss (the same MLM objective used to train BERT, maintaining language modeling capability), and cosine embedding loss (aligning the student's hidden representations with the teacher's, ensuring similar internal representations). DistilBERT's architecture modifications include: reducing the number of transformer layers by half (6 layers instead of BERT-Base's 12), removing the token-type embedding and the pooler layer, and initializing from every other layer of the pre-trained BERT teacher. The result is 66M parameters compared to BERT-Base's 110M. Performance across GLUE benchmark tasks shows DistilBERT retaining 97% of BERT's performance while achieving 60% speedup on CPU inference. This efficiency makes DistilBERT suitable for edge deployment (mobile devices, IoT), real-time applications requiring low latency, cost-sensitive cloud deployments, and scenarios where multiple models must run simultaneously. DistilBERT demonstrated that knowledge distillation is highly effective for transformer compression, inspiring similar distilled versions of other models (DistilGPT-2, DistilRoBERTa, TinyBERT, MobileBERT) and establishing model distillation as a standard technique in the NLP deployment toolkit.