celu

**CELU** (Continuously Differentiable Exponential Linear Unit) is a **modification of ELU that ensures continuous first derivatives** — addressing the non-differentiability of ELU at $x = 0$ when $alpha eq 1$ by using a scaled exponential formulation. **Properties of CELU** - **Formula**: $ ext{CELU}(x) = egin{cases} x & x > 0 \ alpha(exp(x/alpha) - 1) & x leq 0 end{cases}$ - **$C^1$ Smoothness**: Continuously differentiable everywhere, including at $x = 0$, for any $alpha > 0$. - **Parameterized**: $alpha$ controls the saturation value and the smoothness for negative inputs. - **Paper**: Barron (2017). **Why It Matters** - **Mathematical Correctness**: Fixes the differentiability issue of ELU when $alpha eq 1$. - **Optimization**: Smooth activations generally lead to smoother loss landscapes and easier optimization. - **Niche**: Less widely adopted than GELU/Swish but theoretically well-motivated. **CELU** is **the mathematically correct ELU** — ensuring smooth differentiability for any choice of the saturation parameter.