Activation Functions Non-Linearity

# Activation Functions & Non-linearity

## Introduction & Motivation

Activation Functions: introduce non-linearity. ReLU, Sigmoid, Tanh, GELU. Applications: deep networks, improved expressiveness.

Motivation: Enable networks to learn complex patterns.

Applications: Universal approximation, gradient flow.

---

## Core Concepts & Theory

### ReLU Family

Rectified linear units.

### Sigmoid & Tanh

Smooth activation functions.

### GELU & Swish

Modern smooth activations.

### Gating Mechanisms

Adaptive activation scaling.

---

## Mathematical Formulation

ReLU:
$$f(x) = \max(0, x)$$

Leaky ReLU:
$$f(x) = \max(\alpha x, x)$$

GELU:
$$f(x) = x \cdot \Phi(x)$$

Swish:
$$f(x) = x \cdot \sigma(\beta x)$$

---

## Advanced Theory & Extensions

### Mish Activation

Self-regularizing activation.

### GLU Variants

Gated linear units.

### Learnable Activations

Parametrized functions.

---

## Computational Considerations

ReLU: O(1).

Sigmoid/Tanh: O(exp).

Gelu: O(erf).

---

## Practical Implementation Strategies

### Activation Selection

Task-dependent choices.

### Gradient Stability

Vanishing gradient prevention.

### Inplace Operations

Memory efficiency.

---

## Benchmark Datasets & Evaluation

ImageNet: Activation comparison.

CIFAR-10: Small-scale validation.

GLUE: NLP activation effectiveness.

---

## Key Challenges & Limitations

### Dead ReLU

Zero gradient regions.

### Saturation Regions

Sigmoid/Tanh saturation.

### Computational Cost

Expensive operations.

---

## Hyperparameter Tuning

ReLU slope (Leaky): 0.01-0.3.

Swish beta: 0.5-2.0.

GELU approximation: Exact vs. tanh.

---

## Real-World Applications & Case Studies

Vision: ReLU dominance.

NLP: GELU in Transformers.

Reinforcement Learning: Tanh for bounded outputs.

---

## Integration with Other Methods

Activation + normalization for stable training; + initialization for gradient flow.

---

## Summary & Key Takeaways

Activation Functions enable non-linear learning.

Principles:
1. ReLU: Efficient non-linearity.
2. Smooth activations: Better gradients.
3. Gating: Adaptive computation.
4. Stability: Gradient flow preservation.
5. Selection: Architecture-dependent.

---

## Appendix: Practical Labs

### Lab 1: ReLU Variants

import numpy as np

def relu(x):
 return np.maximum(0, x)

def leaky_relu(x, alpha=0.01):
 return np.where(x > 0, x, alpha * x)

def elu(x, alpha=1.0):
 return np.where(x > 0, x, alpha * (np.exp(x) - 1))

np.random.seed(42)
x = np.random.randn(100) * 2
relu_out = relu(x)
leaky_out = leaky_relu(x)
elu_out = elu(x)

assert relu_out.shape == x.shape, "Correct ReLU shape"
assert np.min(leaky_out) >= np.min(x) * 0.01, "Leaky ReLU non-zero"
print("✓ ReLU variants working")

### Lab 2: Sigmoid and Tanh

import numpy as np

def sigmoid(x):
 return 1 / (1 + np.exp(-np.clip(x, -500, 500)))

def tanh(x):
 return np.tanh(x)

np.random.seed(42)
x = np.random.randn(100)
sigmoid_out = sigmoid(x)
tanh_out = tanh(x)

assert np.all((sigmoid_out >= 0) & (sigmoid_out <= 1)), "Sigmoid in [0,1]"
assert np.all((tanh_out >= -1) & (tanh_out <= 1)), "Tanh in [-1,1]"
print("✓ Sigmoid/Tanh working")

### Lab 3: GELU Activation

import numpy as np

def gelu_tanh_approx(x):
 cdf = 0.5 * (1 + np.tanh(np.sqrt(2/np.pi) * (x + 0.044715 * x**3)))
 return x * cdf

def swish(x, beta=1.0):
 sigmoid = 1 / (1 + np.exp(-beta * np.clip(x, -500, 500)))
 return x * sigmoid

np.random.seed(42)
x = np.random.randn(100)
gelu_out = gelu_tanh_approx(x)
swish_out = swish(x)

assert gelu_out.shape == x.shape, "Correct GELU shape"
assert swish_out.shape == x.shape, "Correct Swish shape"
print("✓ GELU/Swish working")

### Lab 4: Activation Gradients

import numpy as np

def relu_gradient(x):
 return (x > 0).astype(float)

def sigmoid_gradient(x):
 s = 1 / (1 + np.exp(-np.clip(x, -500, 500)))
 return s * (1 - s)

def tanh_gradient(x):
 t = np.tanh(x)
 return 1 - t**2

np.random.seed(42)
x = np.random.randn(100)
relu_grad = relu_gradient(x)
sigmoid_grad = sigmoid_gradient(x)
tanh_grad = tanh_gradient(x)

assert np.all((relu_grad == 0) | (relu_grad == 1)), "ReLU gradient binary"
assert np.all(tanh_grad <= 1), "Tanh gradient bounded"
print("✓ Activation gradients working")

---