token dropping

**Token Dropping** is an efficiency optimization technique used in transformer training and Mixture-of-Experts (MoE) architectures where a fraction of input tokens are deliberately excluded from computation during the forward pass to reduce training cost, improve throughput, or handle expert capacity overflow. In MoE models, tokens exceeding expert capacity are dropped; in dense transformers, tokens can be selectively dropped based on importance scores to accelerate training. **Why Token Dropping Matters in AI/ML:** Token dropping provides **significant computational savings** in transformer training and inference by recognizing that not all tokens contribute equally to learning, enabling faster training with minimal quality degradation when implemented carefully. • **MoE overflow dropping** — In Mixture-of-Experts layers, tokens routed to an already-full expert buffer are dropped and passed through the residual connection only; this is a necessary consequence of fixed expert capacity but must be minimized (<1%) to preserve quality • **Importance-based dropping** — Tokens are scored by estimated importance (e.g., attention entropy, gradient magnitude, router confidence) and low-importance tokens skip transformer layers, reducing FLOPs by 25-50% with <1% quality loss on benchmarks • **Random token dropping** — During training, randomly dropping 10-25% of tokens per layer (similar to dropout but at the token level) acts as regularization while reducing computation; recovered at inference for full quality • **Structured dropping** — Dropping tokens in structured patterns (e.g., every Nth token, dropping padding tokens, dropping repeated subword tokens) preserves sequence coherence while reducing sequence length and quadratic attention cost • **Progressive dropping** — Early layers process all tokens while later layers progressively drop more tokens, based on the observation that later layers have increasingly redundant token representations | Method | Drop Rate | FLOPs Savings | Quality Impact | Use Case | |--------|-----------|---------------|----------------|----------| | MoE Overflow | 1-20% | Indirect | Proportional to rate | Expert capacity limits | | Importance Scoring | 25-50% | 25-50% | <1% loss | Training acceleration | | Random (Train) | 10-25% | 10-25% | Regularization benefit | Training efficiency | | Structured | 25-50% | 25-50% | Task-dependent | Long sequence processing | | Progressive | 10-40% per layer | 15-30% total | <0.5% loss | Inference optimization | **Token dropping is a versatile efficiency technique that exploits the redundancy inherent in token sequences to reduce computational cost in transformer training and inference, enabling significant throughput improvements with carefully controlled quality tradeoffs in both dense and MoE architectures.**