softplus

**Softplus** is a **smooth approximation to ReLU defined as $f(x) = ln(1 + e^x)$** — providing a continuously differentiable alternative that never outputs exactly zero, making it useful in contexts where strict positivity is required. **Properties of Softplus** - **Formula**: $ ext{Softplus}(x) = ln(1 + e^x)$ - **Derivative**: $ ext{Softplus}'(x) = sigma(x)$ (the sigmoid function). - **Approximation**: Closely approximates ReLU for large $|x|$. Smoother near zero. - **Strictly Positive**: $ ext{Softplus}(x) > 0$ for all $x$ (unlike ReLU which outputs 0 for $x leq 0$). **Why It Matters** - **Variance Modeling**: Used as the output activation for predicting variance/scale parameters (must be positive). - **Theoretical**: Connects ReLU to sigmoid through differentiation (Softplus → sigmoid → logistic). - **Building Block**: Used inside other activations like Mish: $ ext{Mish}(x) = x cdot anh( ext{Softplus}(x))$. **Softplus** is **the smooth version of ReLU** — a continuously differentiable, strictly positive function used where smoothness and positivity are essential.