activation functions relu sigmoid tanh gelu
# Activation Functions: ReLU, Sigmoid, Tanh & GELU
## Introduction & Motivation
Activation functions: non-linearities enabling deep learning. ReLU: simple, efficient; dead neuron risk. Sigmoid/Tanh: historical; vanishing gradients. GELU: smooth approximation; modern alternative. Swish/Mish: learnable smoothing. Applications: all neural networks; critical for expressiveness and trainability.
Motivation: Linear stacking collapses to single linear transform; activations break linearity, enabling universal approximation.
Applications: Feedforward networks, CNNs, RNNs, transformers.
---
## Core Concepts & Theory
### ReLU (Rectified Linear Unit)
Simple threshold: max(0, x); fast computation; sparse activation.
### Sigmoid/Tanh
Sigmoid: (0, 1) output; classical. Tanh: (-1, 1) output; zero-centered.
### GELU (Gaussian Error Linear Unit)
Smooth approximation; learnable behavior; better for transformers.
---
## Mathematical Formulation
ReLU:
$$f(x) = \max(0, x)$$
Sigmoid:
$$f(x) = \frac{1}{1 + e^{-x}}$$
Tanh:
$$f(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$$
GELU (approximate):
$$f(x) = x \Phi(x) \approx 0.5 x \left[1 + anh\left(\sqrt{\frac{2}{\pi}}(x + 0.044715 x^3)
ight)
ight]$$
---
## Advanced Theory & Extensions
### Leaky ReLU
Non-zero gradient for negative inputs; avoid dead neurons.
### ELU (Exponential Linear Unit)
Smooth for negatives; reduce internal covariate shift.
### Swish/Mish
Self-gated, learnable smoothing; used in EfficientNets, transformers.
---
## Computational Considerations
ReLU: O(1) element-wise; zero overhead.
Sigmoid/Tanh: O(1) element-wise; exp() slower.
GELU: O(1) approximation; similar to Tanh.
---
## Practical Implementation Strategies
### ReLU for Hidden Layers
Standard choice; fast, sparse.
### Sigmoid for Binary Output
Probability output; normalized [0, 1].
### GELU for Transformers
Smooth gradient flow; superior for deep models.
---
## Benchmark Datasets & Evaluation
ImageNet: ReLU standard; ResNets use ReLU.
BERT: GELU default; outperforms ReLU on language tasks.
Deep Networks (>50 layers): GELU/Swish reduce training instability.
---
## Key Challenges & Limitations
### Dead ReLU
Neurons stuck at 0; poor initialization or high learning rates cause.
### Vanishing Gradients
Sigmoid/Tanh: gradients → 0 in saturated region.
### Computational Cost vs Simplicity
GELU more expensive than ReLU; worth it for deep models.
---
## Hyperparameter Tuning
ReLU variants: Leaky ReLU α ∈ [0.01, 0.3].
Output activation: Sigmoid (binary), Softmax (multiclass), Linear (regression).
Depth dependency: ReLU for shallow, GELU for deep (>12 layers).
---
## Real-World Applications & Case Studies
Vision: ResNets, EfficientNets use ReLU; Vision Transformers use GELU.
NLP: BERT, GPT use GELU; outperforms ReLU on transformers.
Audio: Speech recognition; ReLU standard.
---
## Integration with Other Methods
Activation + Batch Norm → reduce dead ReLU problem.
Activation + Residual → enable deep networks.
---
## Summary & Key Takeaways
Activation functions via ReLU, GELU, and variants enable non-linear expressiveness, balancing computational efficiency and gradient flow for deep learning.
Principles:
1. ReLU: efficient, sparse; risk of dead neurons.
2. Sigmoid/Tanh: classical; suffer vanishing gradients.
3. GELU: smooth, learnable; modern standard.
4. Leaky/ELU: mitigate dead neuron problem.
5. Depth-dependent: shallow→ReLU, deep→GELU.
---
---
## Appendix: Practical Labs
### Lab 1: Activation Functions
import torch
import torch.nn.functional as F
import numpy as np
def activation_comparison(x):
"""Compare activation functions"""
relu = F.relu(x)
sigmoid = torch.sigmoid(x)
tanh = torch.tanh(x)
gelu = F.gelu(x)
return {'relu': relu, 'sigmoid': sigmoid, 'tanh': tanh, 'gelu': gelu}
# Test
np.random.seed(42)
x = torch.randn(32, 100)
activations = activation_comparison(x)
assert activations['relu'].shape == (32, 100), "Should preserve shape"
assert (activations['sigmoid'] >= 0).all() and (activations['sigmoid'] <= 1).all(), "Sigmoid in [0,1]"
assert (activations['tanh'] >= -1).all() and (activations['tanh'] <= 1).all(), "Tanh in [-1,1]"
print("✓ Activation functions working")
if __name__ == "__main__":
print("Lab 1: Activations - PASSED")### Lab 2: Gradient Flow
import torch
import torch.nn.functional as F
import numpy as np
def compute_gradients(x, activation_fn):
"""Compute gradient magnitudes through activation"""
x = x.requires_grad_(True)
y = activation_fn(x).sum()
y.backward()
grad_magnitude = x.grad.abs().mean().item()
return grad_magnitude
# Test
np.random.seed(42)
x = torch.randn(100, 50)
grad_relu = compute_gradients(x, F.relu)
grad_sigmoid = compute_gradients(x, torch.sigmoid)
grad_tanh = compute_gradients(x, torch.tanh)
assert grad_relu > 0, "ReLU gradients should be positive"
assert grad_sigmoid > 0, "Sigmoid gradients should be positive"
assert grad_tanh > 0, "Tanh gradients should be positive"
print("✓ Gradient flow working")
if __name__ == "__main__":
print("Lab 2: Gradients - PASSED")### Lab 3: Dead Neuron Analysis
import torch
import torch.nn as nn
import numpy as np
def count_dead_neurons(x, activation_fn):
"""Count neurons producing zero activation"""
activations = activation_fn(x)
dead_count = (activations == 0).sum().item()
total_count = activations.numel()
return dead_count / total_count
# Test
np.random.seed(42)
x = torch.randn(100, 100)
# Simulate negative initialization (causes dead neurons)
x_negative = torch.randn(100, 100) - 2.0
dead_ratio_negative = count_dead_neurons(x_negative, torch.relu)
assert dead_ratio_negative > 0, "Negative input should have dead neurons"
assert dead_ratio_negative < 1, "Not all should be dead"
print("✓ Dead neuron analysis working")
if __name__ == "__main__":
print("Lab 3: Dead Neurons - PASSED")### Lab 4: Activation Properties
import torch
import torch.nn.functional as F
import numpy as np
def activation_properties(x):
"""Measure activation properties: sparsity, saturation"""
relu = F.relu(x)
sigmoid = torch.sigmoid(x)
# Sparsity: fraction of zeros
relu_sparsity = (relu == 0).float().mean().item()
# Saturation: fraction near boundary
sigmoid_saturation = ((sigmoid < 0.1) | (sigmoid > 0.9)).float().mean().item()
return {'relu_sparsity': relu_sparsity, 'sigmoid_saturation': sigmoid_saturation}
# Test
np.random.seed(42)
x = torch.randn(100, 100)
props = activation_properties(x)
assert 0 <= props['relu_sparsity'] <= 1, "Sparsity in [0,1]"
assert 0 <= props['sigmoid_saturation'] <= 1, "Saturation in [0,1]"
assert np.isfinite(props['relu_sparsity']), "Sparsity should be finite"
print("✓ Activation properties working")
if __name__ == "__main__":
print("Lab 4: Properties - PASSED")