rrelu

**RReLU** (Randomized Leaky ReLU) is a **variant of Leaky ReLU where the negative slope is randomly sampled from a uniform distribution during training** — and fixed to the mean of that distribution during inference, providing built-in regularization. **Properties of RReLU** - **Training**: $ ext{RReLU}(x) = egin{cases} x & x > 0 \ a cdot x & x leq 0 end{cases}$ where $a sim U( ext{lower}, ext{upper})$ (typically $U(0.01, 0.33)$). - **Inference**: $a = ( ext{lower} + ext{upper}) / 2$ (deterministic). - **Regularization**: The randomness during training acts as a stochastic regularizer (similar to dropout). - **Paper**: Xu et al. (2015). **Why It Matters** - **Built-In Regularization**: The random slope provides implicit regularization without explicit dropout. - **Kaggle**: Popular in competition settings where every bit of regularization helps. - **Simplicity**: No learnable parameters (unlike PReLU), but with regularization benefits. **RReLU** is **the stochastic ReLU** — introducing randomness in the negative slope for built-in regularization during training.