attention rollout in vit
**Attention rollout in ViT** is the **layer-wise aggregation method that composes attention matrices across depth to estimate end-to-end token influence on final predictions** - instead of viewing one layer in isolation, rollout traces how information propagates from input patches to output tokens.
**What Is Attention Rollout?**
- **Definition**: Recursive multiplication of attention matrices with identity residual terms across transformer layers.
- **Core Idea**: Influence accumulates through many blocks, so global attribution must include the full chain.
- **Output**: A single influence map showing patch contribution to CLS or target token.
- **Scope**: Works for classification and can be adapted to dense token outputs.
**Why Attention Rollout Matters**
- **Deeper Explainability**: Captures cross-layer pathways missed by single-layer heatmaps.
- **Consistency Checks**: Detects if influence remains stable across augmentations and seeds.
- **Bias Detection**: Highlights unintended dependencies on background regions.
- **Model Comparison**: Enables fair explainability comparison across ViT variants.
- **Debugging Efficiency**: Reduces manual review time by summarizing layer dynamics.
**How Rollout Is Computed**
**Step 1**:
- Collect attention matrices A_l from each layer and average or select heads.
- Add identity matrix to model residual mixing, then normalize rows.
**Step 2**:
- Multiply adjusted matrices from shallow to deep layers to obtain cumulative influence matrix.
- Extract influence from output token to input patch tokens.
**Step 3**:
- Reshape influence vector to patch grid and overlay as saliency map.
- Validate map behavior against counterfactual image edits.
**Implementation Notes**
- **Head Aggregation**: Mean aggregation is stable baseline, max can overemphasize outliers.
- **Numerical Stability**: Use float32 for matrix products in long depth models.
- **Residual Handling**: Identity blending choice strongly affects attribution sharpness.
Attention rollout in ViT is **a robust way to summarize multi-layer information flow and patch influence in one interpretable map** - it turns raw attention tensors into actionable explainability signals for model governance.