dropout regularization

**Dropout** — a regularization technique that randomly deactivates neurons during training, forcing the network to learn redundant representations and reducing overfitting. **How It Works** - During training: Each neuron is set to zero with probability $p$ (typically 0.1–0.5) - During inference: All neurons are active, but outputs are scaled by $(1-p)$ to compensate - Effect: The network can't rely on any single neuron — must learn distributed, robust features **Why It Works** - Approximate ensemble: Each training step uses a different sub-network. Dropout is like training $2^n$ networks simultaneously - Prevents co-adaptation: Neurons can't learn to depend on specific partners **Variants** - **Standard Dropout**: Applied to fully connected layers - **Spatial Dropout (Dropout2D)**: Drops entire feature maps in CNNs (more effective than per-pixel) - **DropConnect**: Drops weights instead of activations - **DropPath/Stochastic Depth**: Drops entire residual blocks (used in Vision Transformers) **Practical Tips** - Typically $p=0.5$ for hidden layers, $p=0.1$–$0.2$ for input layers - Don't use with Batch Normalization (they conflict — BN already regularizes) - Always disable during evaluation: `model.eval()` in PyTorch **Dropout** remains one of the most effective and widely-used regularization techniques despite its simplicity.