selu

**SELU** (Scaled Exponential Linear Unit) is a **self-normalizing activation function that automatically maintains zero mean and unit variance of activations** — with specific scale parameters ($lambda approx 1.0507$, $alpha approx 1.6733$) derived to create a fixed-point attractor for the activation statistics. **Properties of SELU** - **Formula**: $ ext{SELU}(x) = lambda egin{cases} x & x > 0 \ alpha(e^x - 1) & x leq 0 end{cases}$ - **Self-Normalizing**: Activations converge to zero mean and unit variance, even without BatchNorm. - **Requires**: Specific initialization (LeCun Normal) and standard feedforward architecture. - **Paper**: Klambauer et al. (2017). **Why It Matters** - **No BatchNorm Needed**: Self-normalization eliminates the need for explicit normalization layers. - **Deep Feedforward**: Enables training 100+ layer feedforward networks without BN. - **Limitation**: Only works well with fully connected architectures and specific initialization. **SELU** is **the self-normalizing activation** — a mathematically designed fixed point that keeps activations stable through arbitrarily deep networks.