activation function zoo

**Activation Function Zoo** refers to the **large and growing collection of activation functions available for neural networks** — from the classic sigmoid and tanh to modern learnable variants like Swish, Mish, and GELU, each with different properties for gradient flow, performance, and computational cost. **The Major Families** - **Classic**: Sigmoid, Tanh — smooth but suffer from vanishing gradients. - **ReLU Family**: ReLU, Leaky ReLU, PReLU, ELU, SELU — fast, sparse, but can die (zero gradients). - **Smooth Non-Saturating**: Swish, Mish, GELU — smooth approximations to ReLU with better gradient properties. - **Learnable**: PReLU, Maxout, PAU — parameters that adapt during training. - **Gated**: GLU, SwiGLU, GeGLU — multiplicative gating for transformers. **Why It Matters** - **Architecture-Dependent**: The best activation varies by architecture (ReLU for CNNs, GELU for transformers, SwiGLU for LLMs). - **Subtle Impact**: Activation choice affects convergence speed, final accuracy, and computational cost. - **No Universal Best**: Despite decades of research, no single activation dominates all settings. **The Activation Zoo** is **the menagerie of nonlinearities** — each species evolved for a different ecological niche in the deep learning ecosystem.