Batch Normalization Variants

# Batch Normalization & Variants

## Introduction & Motivation

Batch normalization: normalize activations per mini-batch. Accelerate training, improve generalization. Applications: deep neural networks, training stability.

Motivation: Reduce internal covariate shift during training.

Applications: Stable training, faster convergence, improved performance.

---

## Core Concepts & Theory

### Normalization

Zero mean, unit variance.

### Learnable Parameters

Scale and shift (gamma, beta).

### Running Statistics

Track batch mean and variance.

### Test-Time Behavior

Use running estimates.

---

## Mathematical Formulation

Batch Normalization:
$$\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$$

Scale and Shift:
$$y_i = \gamma \hat{x}_i + \beta$$

Running Average:
$$\mu_{ ext{running}} = (1-\alpha)\mu_{ ext{running}} + \alpha \mu_B$$

---

## Advanced Theory & Extensions

### Layer Normalization

Normalize across features.

### Group Normalization

Normalize per group.

### Instance Normalization

Per-instance normalization.

---

## Computational Considerations

Training: Normalize per batch.

Inference: Use running statistics.

Memory: Store mean and variance.

---

## Practical Implementation Strategies

### Momentum

Exponential moving average.

### Epsilon

Numerical stability.

### Affine Parameters

Enable scale and shift.

---

## Benchmark Datasets & Evaluation

ImageNet: Vision classification.

CIFAR-10: Smaller datasets.

ResNet: Standard benchmark.

---

## Key Challenges & Limitations

### Batch Dependency

Different behavior train vs test.

### Small Batches

Unstable with tiny batches.

### Synchronization

Distributed training complexity.

---

## Hyperparameter Tuning

Momentum: 0.9-0.99.

Epsilon: 1e-3 to 1e-5.

Affine: Usually True.

---

## Real-World Applications & Case Studies

Deep Networks: Essential for training.

Accelerated Training: 2-5× faster.

Improved Accuracy: Better generalization.

---

## Integration with Other Methods

Batch norm + residual connections; + dropout.

---

## Summary & Key Takeaways

Batch normalization stabilizes deep network training.

Principles:
1. Normalization: Zero mean, unit variance.
2. Learning: Affine parameters adapt.
3. Running stats: Test-time estimates.
4. Acceleration: Faster convergence.
5. Variants: Task-specific choices.

---

## Appendix: Practical Labs

### Lab 1: Batch Normalization Forward

import numpy as np

def batch_norm_forward(x, gamma, beta, eps=1e-5):
 """Batch normalization forward pass"""
 # Compute batch statistics
 mu = np.mean(x, axis=0)
 sigma = np.std(x, axis=0)
 
 # Normalize
 x_hat = (x - mu) / (sigma + eps)
 
 # Scale and shift
 y = gamma * x_hat + beta
 
 return y, mu, sigma

np.random.seed(42)
x = np.random.randn(32, 784)
gamma = np.ones(784)
beta = np.zeros(784)
y, mu, sigma = batch_norm_forward(x, gamma, beta)
assert y.shape == x.shape
print("✓ Batch norm forward working")

### Lab 2: Running Statistics

import numpy as np

def update_running_stats(running_mean, running_var, batch_mean, batch_var, momentum=0.9):
 """Update running statistics"""
 running_mean = momentum * running_mean + (1 - momentum) * batch_mean
 running_var = momentum * running_var + (1 - momentum) * batch_var
 return running_mean, running_var

np.random.seed(42)
run_mean = np.zeros(10)
run_var = np.ones(10)
batch_mean = np.random.randn(10)
batch_var = np.random.rand(10)
run_mean, run_var = update_running_stats(run_mean, run_var, batch_mean, batch_var)
print("✓ Running statistics update working")

### Lab 3: Layer Normalization

import numpy as np

def layer_norm(x, gamma, beta, eps=1e-5):
 """Layer normalization"""
 # Normalize across features
 mu = np.mean(x, axis=-1, keepdims=True)
 sigma = np.std(x, axis=-1, keepdims=True)
 
 x_hat = (x - mu) / (sigma + eps)
 y = gamma * x_hat + beta
 
 return y

np.random.seed(42)
x = np.random.randn(32, 512)
gamma = np.ones(512)
beta = np.zeros(512)
y = layer_norm(x, gamma, beta)
assert y.shape == x.shape
print("✓ Layer normalization working")

### Lab 4: Group Normalization

import numpy as np

def group_norm(x, num_groups=32, gamma=None, beta=None, eps=1e-5):
 """Group normalization"""
 N, C = x.shape[:2]
 x = x.reshape(N, num_groups, C // num_groups)
 
 # Normalize per group
 mu = np.mean(x, axis=(1, 2), keepdims=True)
 sigma = np.std(x, axis=(1, 2), keepdims=True)
 
 x_hat = (x - mu) / (sigma + eps)
 x_hat = x_hat.reshape(N, C)
 
 if gamma is not None:
 x_hat = gamma * x_hat + beta
 
 return x_hat

np.random.seed(42)
x = np.random.randn(8, 256)
y = group_norm(x, num_groups=32)
assert y.shape == x.shape
print("✓ Group normalization working")

---