adaptive activation

**Adaptive activation functions** are **learnable nonlinearities with parameters adjusted during training** — enabling networks to learn optimal activation shapes per layer, including PReLU (learnable slope), Swish (learnable temperature), and other parameterized functions that customize nonlinearities to specific tasks and architectures. **Common Adaptive Activations** - **PReLU**: f(x) = max(αx, x) where α is learned (per-channel or global) - **Swish-β**: f(x) = x * σ(βx) where β is learned scaling - **Maxout**: f(x) = max(w₁x, w₂x, ..., wₖx) — piecewise linear with k slopes - **APL**: Adaptive Piecewise Linear with learned breakpoints **Advantages** - Task-specific optimization — networks learn activation shapes - Often outperform fixed activations on specific domains - Flexible — adapt to requirements of different layers - Learnable — parameters tuned via standard gradient descent Adaptive activations enable **learned-from-data nonlinearities** — networks discover optimal activation shapes.