CELU (Continuously Differentiable Exponential Linear Unit) is a modification of ELU that ensures continuous first derivatives — addressing the non-differentiability of ELU at $x = 0$ when $alpha eq 1$ by using a scaled exponential formulation.

Properties of CELU

• Formula: $ ext{CELU}(x) = egin{cases} x & x > 0 \ alpha(exp(x/alpha) - 1) & x leq 0 end{cases}$
• $C^1$ Smoothness: Continuously differentiable everywhere, including at $x = 0$, for any $alpha > 0$.
• Parameterized: $alpha$ controls the saturation value and the smoothness for negative inputs.
• Paper: Barron (2017).

Why It Matters

• Mathematical Correctness: Fixes the differentiability issue of ELU when $alpha

eq 1$.

• Optimization: Smooth activations generally lead to smoother loss landscapes and easier optimization.
• Niche: Less widely adopted than GELU/Swish but theoretically well-motivated.

CELU is the mathematically correct ELU — ensuring smooth differentiability for any choice of the saturation parameter.