Adaptive activation functions are learnable nonlinearities with parameters adjusted during training — enabling networks to learn optimal activation shapes per layer, including PReLU (learnable slope), Swish (learnable temperature), and other parameterized functions that customize nonlinearities to specific tasks and architectures.

Common Adaptive Activations

• PReLU: f(x) = max(αx, x) where α is learned (per-channel or global)
• Swish-β: f(x) = x * σ(βx) where β is learned scaling
• Maxout: f(x) = max(w₁x, w₂x, ..., wₖx) — piecewise linear with k slopes
• APL: Adaptive Piecewise Linear with learned breakpoints

Advantages

• Task-specific optimization — networks learn activation shapes
• Often outperform fixed activations on specific domains
• Flexible — adapt to requirements of different layers
• Learnable — parameters tuned via standard gradient descent

Adaptive activations enable learned-from-data nonlinearities — networks discover optimal activation shapes.