RReLU (Randomized Leaky ReLU) is a variant of Leaky ReLU where the negative slope is randomly sampled from a uniform distribution during training — and fixed to the mean of that distribution during inference, providing built-in regularization.

Properties of RReLU

• Training: $ ext{RReLU}(x) = egin{cases} x & x > 0 \ a cdot x & x leq 0 end{cases}$ where $a sim U( ext{lower}, ext{upper})$ (typically $U(0.01, 0.33)$).
• Inference: $a = ( ext{lower} + ext{upper}) / 2$ (deterministic).
• Regularization: The randomness during training acts as a stochastic regularizer (similar to dropout).
• Paper: Xu et al. (2015).

Why It Matters

• Built-In Regularization: The random slope provides implicit regularization without explicit dropout.
• Kaggle: Popular in competition settings where every bit of regularization helps.
• Simplicity: No learnable parameters (unlike PReLU), but with regularization benefits.

RReLU is the stochastic ReLU — introducing randomness in the negative slope for built-in regularization during training.