Online Distillation and Co-Distillation is the training paradigm where multiple neural networks teach each other simultaneously during training — unlike traditional knowledge distillation where a pre-trained large teacher transfers knowledge to a smaller student, online distillation trains teacher and student (or multiple peers) jointly from scratch, enabling mutual improvement where networks with different architectures or capacities share complementary knowledge through soft label exchange, logit matching, and feature alignment without requiring a separately trained teacher model.

Traditional vs. Online Distillation

Traditional (Offline) Distillation:
  Step 1: Train large teacher to convergence
  Step 2: Freeze teacher → train student on teacher's soft labels
  Cost: 2× training time (teacher + student)

Online (Co-)Distillation:
  Step 1: Train all networks simultaneously
  Each network is both teacher AND student
  Cost: ~1.3× training a single network (parallel)

Key Approaches

Deep Mutual Learning

# Two networks training together
for batch in dataloader:
    logits_1 = model_1(batch)
    logits_2 = model_2(batch)
    
    # Standard CE loss for both
    loss_ce_1 = cross_entropy(logits_1, labels)
    loss_ce_2 = cross_entropy(logits_2, labels)
    
    # Mutual KL divergence (each teaches the other)
    loss_kl_1 = kl_div(log_softmax(logits_1/T), softmax(logits_2/T)) * T*T
    loss_kl_2 = kl_div(log_softmax(logits_2/T), softmax(logits_1/T)) * T*T
    
    # Combined losses
    loss_1 = loss_ce_1 + alpha * loss_kl_1
    loss_2 = loss_ce_2 + alpha * loss_kl_2

Why Does Mutual Learning Work?

• Different random initializations → different local features learned.
• Each model discovers patterns the other missed → knowledge complementarity.
• Soft labels provide richer training signal than hard one-hot labels.
• Dark knowledge: The relative probabilities of incorrect classes carry information about data structure.
• Result: Both models end up better than either would alone — even equally-sized peers improve each other.

Self-Distillation

• Add auxiliary classifiers at intermediate layers.
• Deep layers' soft predictions train shallow layers.
• At inference, use only the final layer (no overhead).
• Surprisingly: Even the deepest layer improves from teaching shallower ones.

Applications

Online distillation is the efficient alternative to traditional teacher-student training — by eliminating the need for a separately pre-trained teacher and enabling networks to improve each other during joint training, co-distillation reduces total training cost while often achieving better accuracy than offline distillation, making it particularly valuable when training large teacher models is impractical or when mutual knowledge exchange between diverse model architectures is desired.