layerscale
**LayerScale** is the **trainable scaling factor that fades each block's residual updates at initialization so very deep Vision Transformers remain stable** — initializing the scale to a tiny value (e.g., 1e-4) makes the block behave like identity early on and gradually lets the network grow complexity as training converges.
**What Is LayerScale?**
- **Definition**: A per-channel learnable parameter that multiplies the output of the attention or feed-forward sublayers before adding the residual connection.
- **Key Feature 1**: Scale parameters start small, preventing the residual path from dominating before the block learns useful transformations.
- **Key Feature 2**: LayerScale can be applied to attention outputs, MLP outputs, or both, giving architects flexibility.
- **Key Feature 3**: Because the parameters are trainable, the model learns when to amplify each block as training progresses.
- **Key Feature 4**: Works hand-in-hand with Pre-LN to keep gradients flowing through identity paths.
**Why LayerScale Matters**
- **Gradient Stability**: Early in training, residual contributions are tiny, so the identity path carries gradients without exploding.
- **Deep Models**: Enables stable training of 100-1,000 layer transformers by localizing adjustments per block.
- **Adaptation**: Blocks learn to trust their own transformations only when they become confident.
- **Compatibility**: LayerScale is lightweight (one scalar per channel) and incurs minimal overhead.
- **Calibration**: Prevents sudden spikes in activation magnitude that can destabilize normalization layers.
**Scale Placement**
**Attention Scaling**:
- Multiply the attention output by LayerScale before the residual addition.
- Helps prevent attention from overpowering the signal early in training.
**MLP Scaling**:
- Similarly scale the feed-forward output to avoid immediate large activations.
- Most effective when both sublayers use LayerScale.
**Per-Head Variation**:
- Assign distinct scales per attention head for finer control over each head's contribution.
**How It Works / Technical Details**
**Step 1**: Apply a learnable diagonal matrix (scale factor per channel) to the output of the sublayer before adding the residual connection.
**Step 2**: During backpropagation, the scale parameters adjust so that blocks can gradually emerge from near-identity behavior to full expressivity without destabilizing the network.
**Comparison / Alternatives**
| Aspect | LayerScale | No Scaling | LayerNorm Tuning |
|--------|------------|------------|-----------------|
| Stability | High | Medium | Medium
| Parameters | Per-channel | None | Per-layer
| Expressivity | Adaptive | Fixed | Fixed
| Implementation | Simple | Simple | Slightly complex
**Tools & Platforms**
- **timm**: Supports LayerScale scalars via `layer_scale_init_value` for ViT and Swin.
- **Hugging Face**: Some ViT configs set LayerScale to avoid training collapse.
- **PyTorch**: Custom modules easily implement per-channel scaling with `nn.Parameter`.
- **Monitoring**: Track scale growth during training to ensure blocks acclimate.
LayerScale is **the tiny multiplier that keeps transformer blocks behaving until they learn something worth adding** — it lets Vision Transformers grow deep without the instability that usually trips up residual stacks.