early exit
**Early Exit** is an adaptive inference optimization technique for deep neural networks where computation terminates at an intermediate layer when a confidence criterion is met, rather than propagating through all layers. Each potential exit point includes a lightweight classifier head that evaluates whether the current representation is sufficiently confident for the final prediction, enabling easier inputs to be processed with fewer layers and lower latency.
**Why Early Exit Matters in AI/ML:**
Early exit provides **input-adaptive computation** that reduces average inference latency and energy consumption by allocating fewer computational resources to simpler inputs while preserving full model capacity for difficult examples.
• **Confidence-based termination** — At each exit point, a classifier head produces a prediction and confidence score (e.g., max softmax probability, entropy); if confidence exceeds a threshold, computation stops and the intermediate prediction is returned
• **Dynamic depth** — Different inputs traverse different numbers of layers: simple, unambiguous inputs may exit after 2-3 layers while complex, ambiguous inputs use the full network depth, optimizing average compute per input
• **Exit ramp design** — Exit classifiers are typically lightweight (linear layer + softmax) attached every N layers (e.g., every 3 layers in a 12-layer BERT); they must be accurate yet cheap to avoid overhead exceeding savings
• **Training strategies** — Joint training with weighted losses at each exit point (early exits weighted lower) ensures all exits produce valid predictions; alternatively, self-distillation from the final layer teaches early exits to approximate full-model behavior
• **Latency-quality tradeoff** — Adjusting the confidence threshold controls the exit distribution: lower thresholds exit earlier (faster, slightly less accurate) while higher thresholds push more inputs to deeper layers (slower, more accurate)
| Configuration | Avg. Exit Layer | Speedup | Quality Impact |
|--------------|----------------|---------|----------------|
| Aggressive (low threshold) | 3-4 of 12 | 3-4× | -1-2% accuracy |
| Balanced | 5-7 of 12 | 1.5-2× | <0.5% loss |
| Conservative (high threshold) | 8-10 of 12 | 1.1-1.3× | Negligible |
| Input-adaptive | Varies per input | 1.5-3× | <0.3% loss |
| With distillation | Earlier avg. | 2-3× | <0.5% loss |
**Early exit is a powerful inference optimization that provides input-adaptive computation depth, enabling transformer and deep network models to process simple inputs with a fraction of the full model's computational cost while maintaining high accuracy through confidence-calibrated dynamic termination at intermediate layers.**