dynamic token pruning

**Dynamic Token Pruning** is a **token pruning approach where the pruning decisions are made dynamically at each layer based on learned criteria** — allowing different layers to prune different tokens, and different inputs to have different pruning patterns. **How Does Dynamic Token Pruning Work?** - **Per-Layer Decision**: At each layer, a lightweight predictor determines which tokens to keep. - **Progressive**: Early layers may keep most tokens; later layers prune more aggressively. - **Learned Pruning**: The pruning predictor is trained jointly with the main network (Gumbel-softmax or straight-through estimator). - **Example**: DynamicViT uses a prediction module trained with a distillation loss. **Why It Matters** - **Input-Adaptive**: Easy images prune many tokens early. Complex images retain more tokens longer. - **Layer-Adaptive**: Different layers can focus on different tokens — earlier layers keep diverse tokens, later layers keep only task-relevant ones. - **Accuracy**: Trained pruning predictors maintain accuracy better than heuristic pruning methods. **Dynamic Token Pruning** is **learned selective attention** — training the model to automatically decide which tokens to keep at each layer for optimal efficiency.