Activation Functions Survey (ReLU, GELU, SiLU) compares fundamental non-linearities used in deep learning that introduce non-linearity enabling neural networks to learn complex functions — each activation offering different trade-offs in complexity, gradient flow, and computational efficiency across modern architectures from CNNs to transformers.

ReLU (Rectified Linear Unit):

• Formula: ReLU(x) = max(0, x) — identity for x>0, zero for x≤0
• History: introduced in 2011, revolutionized deep learning enabling efficient training of very deep networks (AlexNet, ResNet)
• Advantages: computationally simple (single comparison), sparse activation (50% neurons inactive) providing implicit regularization
• Gradient: ∂ReLU/∂x = 0 for x<0, 1 for x>0 — clean gradients but zero gradients for negative inputs (dying ReLU problem)
• Dying ReLU: neurons with negative pre-activations permanently zero out in some settings — particularly in early training with poor initialization

ReLU Variants and Extensions:

• Leaky ReLU: ReLU(x) = x if x>0 else 0.01×x — small negative slope prevents complete zero-out, improves gradient flow
• ELU (Exponential Linear Unit): ELU(x) = x if x>0 else α(e^x - 1) — smooth exponential transition, reduces mean shift effect
• SELU (Scaled ELU): adding scale factors enabling self-normalization — maintains unit mean/variance across layers without explicit normalization
• Gelu (Gaussian Error Linear Unit): Gelu(x) = x·Φ(x) where Φ is standard normal CDF — smooth approximation to ReLU with superior gradient flow

GELU (Gaussian Error Linear Unit):

• Formula: Gelu(x) = x·Φ(x) ≈ 0.5x(1 + tanh(√(2/π)(x + 0.044715x³))) — approximation formula enables efficient computation
• Motivation: modeling stochasticity as Gaussian; GELU(x) = x·P(X≤x) for X~N(0,1)
• Characteristics: smooth everywhere with non-zero gradient even for negative inputs — eliminates dying unit problem
• Adoption: standard in BERT, GPT-2, RoBERTa; chosen for superior downstream task performance vs ReLU
• Computational Cost: slightly higher than ReLU due to approximation formula; negligible overhead on modern hardware

GELU vs ReLU Empirical Comparison:

• Perplexity: GELU achieves 3.2 on WIKITEXT-103 vs ReLU 3.5 — consistently better language modeling
• Fine-tuning: BERT-base with GELU outperforms ReLU by 1-2% on GLUE tasks — consistent across task types
• Training Convergence: GELU enables slightly faster convergence (fewer steps to same loss) due to better gradient flow
• Computational Speed: GELU marginally slower per-step but reaches target performance in fewer steps — net training time comparable

SiLU (Swish, Sigmoid Linear Unit):

• Formula: SiLU(x) = x·sigmoid(x) — self-gated with sigmoid controlling magnitude based on input
• Characteristics: smooth everywhere, non-zero gradient for all inputs (no dying units), exhibits interesting saturating properties
• Gating Intuition: sigmoid(x) acts as soft gate (0.5 at zero, approaching 0/1 for extreme values) — enables learned importance weighting
• Adoption: used in EfficientNet, mobile architectures; becoming standard for modern LLMs (Llama, PaLM use SiLU variants)
• Performance: SiLU typically matches or exceeds GELU with slightly lower computational overhead

Modern Activation Function Trends:

• Transformer Standard: GELU or SiLU becoming default in transformers; ReLU deprecated for new architectures
• Parameter Efficiency: SiLU enables more efficient parameter utilization — lower parameter models with SiLU outperform higher-param ReLU models
• Scaling Laws: activation function choice influences scaling laws; SiLU/GELU models scale more efficiently with compute
• Hardware Alignment: modern GPUs optimize GELU/SiLU similarly to ReLU — no practical speed penalty for superior activation

Gradient Flow Characteristics:

• Gradient Magnitude: ReLU gradient 0 or 1 (sharp); GELU/SiLU have smooth gradients varying continuously — less vanishing gradient risk
• Second Derivative: GELU/SiLU have non-zero second derivatives enabling better curvature information for optimization
• Initialization Interaction: better gradient flow enables He initialization without fine-tuning for GELU/SiLU vs ReLU
• Deep Network Training: >50 layer networks train more stably with GELU/SiLU vs ReLU — evident in modern architectures

Activation Statistics and Learned Representations:

• Sparsity: ReLU induces 50% sparsity naturally; GELU/SiLU less sparse (80-90% active) — affects model capacity/efficiency trade-offs
• Information Content: analyzing mutual information between activations and outputs; GELU/SiLU preserve more information than ReLU
• Saturation: tanh/sigmoid saturate for |x|>2; GELU/SiLU less prone to saturation — enables better gradient flow
• Dead Neuron Rate: ReLU 5-15% dead neurons depending on initialization; GELU/SiLU <1% dead neurons

Computational Complexity and Hardware Considerations:

• FLOPS: ReLU 1 comparison operation (essentially free); GELU 2-3 FLOPs; SiLU 3 FLOPs (sigmoid + multiply)
• Peak Throughput: A100 tensor cores achieve >90% peak FLOPS for matrix multiply regardless of activation (overhead <5%)
• Memory Bandwidth: activation computation negligible compared to matrix multiply; bandwidth-bound operations dominate
• Mobile Devices: ReLU slightly preferred on limited hardware; GELU/SiLU sufficient with modern optimization

Activation Function Selection by Task:

• Image Classification: GELU achieves 1-2% improvement over ReLU on ImageNet; SiLU comparable to GELU
• Language Modeling: GELU/SiLU clearly superior to ReLU (2-4% improvement); standard in modern LLMs
• Object Detection: ReLU still common in detector backbones; GELU increasingly adopted in newer architectures
• Reinforcement Learning: ReLU traditional; GELU emerging as better choice for policy/value networks

Theoretical Understanding:

• Expressiveness: continuous smooth activations (GELU, SiLU) theoretically more expressive than piecewise linear (ReLU)
• Universal Approximation: all smooth activations enable universal approximation given sufficient neurons — theoretical advantages marginal
• Optimization Landscape: GELU/SiLU produce smoother loss landscapes — fewer local minima, easier optimization
• Implicit Regularization: ReLU sparsity provides regularization; GELU/SiLU require explicit regularization (dropout, weight decay)

Activation Functions Survey (ReLU, GELU, SiLU) reveals fundamental shifts in modern architecture design — transitioning from ReLU's computational simplicity to GELU/SiLU's superior optimization properties enabling more efficient scaling of deep networks.