Softplus is a smooth approximation to ReLU defined as $f(x) = ln(1 + e^x)$ — providing a continuously differentiable alternative that never outputs exactly zero, making it useful in contexts where strict positivity is required.

Properties of Softplus

• Formula: $ ext{Softplus}(x) = ln(1 + e^x)$
• Derivative: $ ext{Softplus}'(x) = sigma(x)$ (the sigmoid function).
• Approximation: Closely approximates ReLU for large $|x|$. Smoother near zero.
• Strictly Positive: $ ext{Softplus}(x) > 0$ for all $x$ (unlike ReLU which outputs 0 for $x leq 0$).

Why It Matters

• Variance Modeling: Used as the output activation for predicting variance/scale parameters (must be positive).
• Theoretical: Connects ReLU to sigmoid through differentiation (Softplus → sigmoid → logistic).
• Building Block: Used inside other activations like Mish: $ ext{Mish}(x) = x cdot anh( ext{Softplus}(x))$.

Softplus is the smooth version of ReLU — a continuously differentiable, strictly positive function used where smoothness and positivity are essential.