Student-teacher learning trains a smaller student model to mimic a larger teacher model's behavior — enabling deployment of compact, efficient models that retain much of the teacher's capability through knowledge distillation, intermediate layer matching, and response imitation.

What Is Student-Teacher Learning?

• Definition: Transfer knowledge from large (teacher) to small (student).
• Goal: Smaller model with similar performance.
• Methods: Logit matching, feature distillation, response copying.
• Applications: Compression, deployment, efficient inference.

Why Student-Teacher

• Deployment: Large models too expensive for production.
• Latency: Small models respond faster.
• Cost: Reduce serving compute costs.
• Edge: Enable on-device inference.
• Efficiency: Better than training small models from scratch.

Training Approaches

Offline Distillation:

1. Train teacher model (or use pretrained)
2. Freeze teacher weights
3. Train student to match teacher

Pro: Stable, simple
Con: Fixed teacher, can't adapt

Online Distillation:

1. Train teacher and student simultaneously
2. Student learns from evolving teacher
3. Sometimes mutual: both learn from each other

Pro: Adaptive, can exceed static teacher
Con: Complex, harder to optimize

Self-Distillation:

1. Model distills to itself (deeper to shallower)
2. Or current model teaches previous version

Pro: No separate teacher needed
Con: Limited knowledge source

Implementation

Complete Training Loop:

import torch
import torch.nn as nn
import torch.nn.functional as F

class StudentTeacherTrainer:
    def __init__(self, teacher, student, temperature=4.0, alpha=0.5):
        self.teacher = teacher.eval()  # Freeze teacher
        self.student = student
        self.temperature = temperature
        self.alpha = alpha
        self.optimizer = torch.optim.AdamW(student.parameters(), lr=1e-4)
    
    def distillation_loss(self, student_logits, teacher_logits, labels):
        # Soft loss (match teacher distribution)
        soft_targets = F.softmax(teacher_logits / self.temperature, dim=-1)
        soft_student = F.log_softmax(student_logits / self.temperature, dim=-1)
        soft_loss = F.kl_div(soft_student, soft_targets, reduction="batchmean")
        soft_loss *= self.temperature ** 2
        
        # Hard loss (match true labels)
        hard_loss = F.cross_entropy(student_logits, labels)
        
        return self.alpha * hard_loss + (1 - self.alpha) * soft_loss
    
    def train_step(self, inputs, labels):
        # Teacher inference (no gradients)
        with torch.no_grad():
            teacher_logits = self.teacher(inputs)
        
        # Student forward pass
        student_logits = self.student(inputs)
        
        # Compute loss
        loss = self.distillation_loss(student_logits, teacher_logits, labels)
        
        # Backprop
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()
        
        return loss.item()

Feature Distillation:

class FeatureDistillationLoss(nn.Module):
    def __init__(self, student_dims, teacher_dims):
        super().__init__()
        # Projectors to match dimensions
        self.projectors = nn.ModuleList([
            nn.Linear(s_dim, t_dim) 
            for s_dim, t_dim in zip(student_dims, teacher_dims)
        ])
    
    def forward(self, student_features, teacher_features):
        loss = 0
        for proj, s_feat, t_feat in zip(
            self.projectors, student_features, teacher_features
        ):
            # Project student to teacher dimension
            s_proj = proj(s_feat)
            # MSE loss between features
            loss += F.mse_loss(s_proj, t_feat)
        return loss

LLM Distillation

Response-Based (Common for LLMs):

def distill_llm(teacher, student, prompts):
    for prompt in prompts:
        # Teacher generates response
        with torch.no_grad():
            teacher_response = teacher.generate(
                prompt, 
                max_tokens=512,
                temperature=0.7
            )
        
        # Student learns to generate same response
        student_loss = student.forward(
            input_ids=prompt + teacher_response,
            labels=teacher_response  # Predict teacher's tokens
        )
        
        student_loss.backward()
        optimizer.step()

Token-Level Matching:

# Match next-token probabilities
student_logits = student(input_ids).logits
teacher_logits = teacher(input_ids).logits

# KL divergence at each position
loss = kl_div(
    log_softmax(student_logits / T),
    softmax(teacher_logits / T)
) * T²

Model Size Guidelines

Teacher Size    | Student Size    | Expected Retention
----------------|-----------------|--------------------
70B parameters  | 7B              | 85-95% quality
7B parameters   | 1.3B            | 80-90% quality
1.3B parameters | 350M            | 75-85% quality

Architecture Choices:

Option 1: Same architecture, fewer layers
Option 2: Same architecture, smaller hidden dim
Option 3: Different architecture entirely

Best: Student architecture matches task needs

Best Practices

Practice              | Recommendation
----------------------|----------------------------------
Data                  | Use teacher's training data if possible
Temperature           | Start with T=4, tune
Training time         | 1-3× normal epochs
Learning rate         | Lower than training from scratch
Label smoothing       | Often redundant with soft targets
Intermediate layers   | Match if architectures similar

Student-teacher learning is the primary method for deploying powerful models efficiently — by transferring knowledge from expensive-to-run teachers to compact students, organizations can deliver AI capabilities at a fraction of the inference cost.