PReLU (Parametric Rectified Linear Unit) is a learnable activation function that extends Leaky ReLU by treating the negative slope coefficient as a trainable parameter learned by backpropagation alongside the network weights — allowing each channel or neuron to adaptively determine how much signal to pass for negative inputs rather than using a fixed, manually chosen leak rate — introduced by Kaiming He et al. (Microsoft Research, 2015) in the same paper as the He weight initialization and directly enabling the training of the deep residual networks that achieved superhuman performance on ImageNet classification, establishing PReLU as the activation function that unlocked the era of very deep convolutional networks.

What Is PReLU?

• Formula: PReLU(x) = x for x > 0; PReLU(x) = a × x for x ≤ 0, where a is a learned scalar parameter.
• Learnable Negative Slope: Unlike standard ReLU (a = 0) and Leaky ReLU (a = fixed small constant, typically 0.01), PReLU's a is a free parameter that gradient descent adjusts during training.
• Per-Channel Parameters: In convolutional networks, PReLU typically uses one a per feature map channel — adding negligible parameters (a few hundred scalars for an entire ResNet) with minimal memory overhead.
• Backpropagation: The gradient with respect to a is simply the sum of all negative input values in that channel — a well-behaved, non-sparse gradient signal.

PReLU vs. Other Activation Functions

The He et al. 2015 Paper: Why PReLU Mattered

The introduction of PReLU was inseparable from two other key contributions in the same paper:

• He Initialization: Proper variance scaling for ReLU networks — ensures signal neither explodes nor vanishes through depth, enabling training >20-layer networks.
• PReLU Activation: With He init + PReLU, the authors trained a 22-layer VGG-style network that surpassed human-level performance on ImageNet for the first time (top-5 error 4.94% vs. human 5.1%).
• ResNets (companion paper): PReLU's ability to pass negative-input gradient without vanishing complemented the skip connections in residual networks, helping train 100+ layer networks.

PReLU's learned a values after training are informative: in early layers they tend to be near zero (ReLU-like — sparse features preferred), while in deeper layers they take larger values (more gradient flow needed to avoid dying channels in deep networks).

When to Use PReLU

• Deep CNNs: Especially effective in image classification networks deeper than 10 layers where dying ReLU channels are a training stability risk.
• Generative Models: GANs and VAEs benefit from full gradient flow to generators — PReLU's nonzero negative slope prevents the generator from having unsupported dead channels.
• Attention-Free Architectures: In networks without layer normalization or residual connections, PReLU's adaptive slope helps stabilize gradient propagation.

PReLU is the activation function that adapts itself to the data — the minimal learnable extension of ReLU that preserves its computational simplicity while allowing each network layer to discover the optimal balance between sparsity and gradient flow, a small but critical contribution to the arsenal of tools that enabled the deep learning revolution in computer vision.