Batch Normalization Layer Normalization Internal Covariate Shift Reduction

# Batch Normalization & Layer Normalization: Internal Covariate Shift Reduction

## Introduction & Motivation

Batch Normalization (BN): normalize activations per batch; reduce internal covariate shift. Layer Normalization (LN): normalize per sample; stable across batch sizes. Group Norm, Instance Norm variants. Applications: deep networks, transformers, RNNs. Improves training stability, convergence speed, generalization.

Motivation: Deep networks suffer from covariate shift; distributions change per layer. Normalization stabilizes; enables higher learning rates.

Applications: Convolutional networks (BN standard), Transformers (LN preferred), RNNs (Layer Norm).

---

## Core Concepts & Theory

### Batch Normalization

Per-batch statistics; normalize to zero mean, unit variance. Learnable affine parameters (scale, shift).

### Layer Normalization

Per-sample statistics; normalize across features. No batch dependence; stable inference.

### Internal Covariate Shift

Distribution changes of layer inputs; slows learning.

---

## Mathematical Formulation

Batch Normalization (training):
$$\hat{x} = \frac{x - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$$
$$y = \gamma \hat{x} + \beta$$

Layer Normalization:
$$\hat{x} = \frac{x - \mu_L}{\sqrt{\sigma_L^2 + \epsilon}}$$
$$y = \gamma \hat{x} + \beta$$

where μ_B = batch mean, σ_B = batch std; μ_L = layer mean (per sample).

---

## Advanced Theory & Extensions

### Batch Renormalization

Modify batch norm to be less dependent on batch statistics.

### Weight Normalization

Reparameterize weights; separate magnitude and direction.

### Instance Normalization

Per-channel per-sample; used in style transfer.

---

## Computational Considerations

BN: O(NHW) per batch; batch dependent (inference differs from training).

LN: O(NHW) per sample; same behavior train/test.

Memory: Batch statistics stored for backward pass.

---

## Practical Implementation Strategies

### Batch Size Dependence

Small batch → noisy estimates; use momentum for EMA.

### Test Time Behavior

BN uses running statistics; accumulate during training.

LN: Identical train/test; no issue.

### Layer Placement

Typically: Linear → Norm → Activation.

---

## Benchmark Datasets & Evaluation

ImageNet (CNN): BN standard; 1-2% accuracy improvement.

Transformer Models: LN preferred; more stable.

Metrics: Convergence speed, final accuracy, training stability.

---

## Key Challenges & Limitations

### Batch Size Sensitivity

BN performs poorly with small batches; LN more robust.

### Train-Test Mismatch

BN statistics differ train/test; can hurt transfer learning.

### Computational Overhead

Small overhead; but adds complexity.

---

## Hyperparameter Tuning

Momentum (BN): 0.9-0.99; EMA weight for running stats.

Epsilon: 1e-5 to 1e-3; numerical stability.

Learnable affine: True (benefits learning).

---

## Real-World Applications & Case Studies

ResNet: BN enables deep networks; >50 layers standard.

BERT/Transformers: LN standard; enables scaling to billions parameters.

Style Transfer: Instance Norm standard for image generation.

---

## Integration with Other Methods

BN + Skip Connections → enable very deep networks.

LN + Attention → stable attention weight learning.

---

## Summary & Key Takeaways

Normalization via Batch or Layer Norm reduces internal covariate shift, improving training stability and enabling deeper, faster-converging networks.

Principles:
1. Batch Norm: batch statistics; train/test difference.
2. Layer Norm: sample statistics; no batch dependence.
3. Learnable affine: γ, β parameters restore expressiveness.
4. Placement: before or after activation (debate).
5. Momentum: EMA for running statistics (BN).

---

---

## Appendix: Practical Labs

### Lab 1: Batch Normalization

import torch
import torch.nn as nn

class BatchNorm1D(nn.Module):
 def __init__(self, num_features, momentum=0.9, eps=1e-5):
 super().__init__()
 self.num_features = num_features
 self.momentum = momentum
 self.eps = eps
 
 self.gamma = nn.Parameter(torch.ones(num_features))
 self.beta = nn.Parameter(torch.zeros(num_features))
 
 self.register_buffer('running_mean', torch.zeros(num_features))
 self.register_buffer('running_var', torch.ones(num_features))
 
 def forward(self, x):
 if self.training:
 batch_mean = x.mean(dim=0)
 batch_var = x.var(dim=0, unbiased=False)
 
 # Update running statistics
 self.running_mean.data = self.momentum * self.running_mean + (1 - self.momentum) * batch_mean
 self.running_var.data = self.momentum * self.running_var + (1 - self.momentum) * batch_var
 
 x_norm = (x - batch_mean) / torch.sqrt(batch_var + self.eps)
 else:
 x_norm = (x - self.running_mean) / torch.sqrt(self.running_var + self.eps)
 
 return self.gamma * x_norm + self.beta

# Test
bn = BatchNorm1D(num_features=10)
x = torch.randn(32, 10)

bn.train()
y_train = bn(x)
assert y_train.mean().abs() < 0.1, "Should normalize mean"

bn.eval()
y_test = bn(x)
assert y_test.shape == x.shape, "Should preserve shape"
print("✓ Batch Norm working")

if __name__ == "__main__":
 print("Lab 1: BN - PASSED")

### Lab 2: Layer Normalization

import torch
import torch.nn as nn

class LayerNorm(nn.Module):
 def __init__(self, num_features, eps=1e-5):
 super().__init__()
 self.gamma = nn.Parameter(torch.ones(num_features))
 self.beta = nn.Parameter(torch.zeros(num_features))
 self.eps = eps
 
 def forward(self, x):
 # Normalize across features (last dimension)
 mean = x.mean(dim=-1, keepdim=True)
 var = x.var(dim=-1, keepdim=True, unbiased=False)
 
 x_norm = (x - mean) / torch.sqrt(var + self.eps)
 return self.gamma * x_norm + self.beta

# Test
ln = LayerNorm(num_features=10)
x = torch.randn(32, 10)

y = ln(x)
assert y.shape == x.shape, "Should preserve shape"
assert y.mean(dim=-1).abs().max() < 0.1, "Should normalize per sample"
print("✓ Layer Norm working")

if __name__ == "__main__":
 print("Lab 2: LN - PASSED")

### Lab 3: BN vs LN Stability

import torch
import torch.nn as nn

def test_norm_stability(batch_sizes):
 """Compare BN and LN with different batch sizes"""
 results = []
 
 for bs in batch_sizes:
 x = torch.randn(bs, 100)
 
 # Batch Norm
 bn = nn.BatchNorm1d(100)
 bn.train()
 y_bn = bn(x)
 bn_std = y_bn.std(dim=0).mean()
 
 # Layer Norm
 ln = nn.LayerNorm(100)
 y_ln = ln(x)
 ln_std = y_ln.std(dim=0).mean()
 
 results.append({'batch_size': bs, 'bn_std': bn_std.item(), 'ln_std': ln_std.item()})
 
 return results

# Test
batch_sizes = [1, 4, 32, 128]
results = test_norm_stability(batch_sizes)

print("Normalization stability:")
for r in results:
 print(f" Batch {r['batch_size']}: BN_std={r['bn_std']:.3f}, LN_std={r['ln_std']:.3f}")
assert len(results) == 4, "Should have 4 results"
print("✓ Stability comparison working")

if __name__ == "__main__":
 print("Lab 3: Stability - PASSED")

### Lab 4: Normalization Effects

import torch
import torch.nn as nn
import numpy as np

def measure_covariate_shift(model_layers, x):
 """Measure distribution shift across layers"""
 
 shifts = []
 for i, layer in enumerate(model_layers):
 x = layer(x)
 shift = x.std().item()
 shifts.append(shift)
 
 return shifts

# Test: model without normalization
model_no_norm = nn.Sequential(
 nn.Linear(10, 50),
 nn.ReLU(),
 nn.Linear(50, 50),
 nn.ReLU(),
 nn.Linear(50, 10)
)

# Model with normalization
model_with_norm = nn.Sequential(
 nn.Linear(10, 50),
 nn.LayerNorm(50),
 nn.ReLU(),
 nn.Linear(50, 50),
 nn.LayerNorm(50),
 nn.ReLU(),
 nn.Linear(50, 10)
)

x = torch.randn(32, 10)
shifts_no_norm = measure_covariate_shift(list(model_no_norm), x.clone())
shifts_with_norm = measure_covariate_shift(list(model_with_norm), x.clone())

print(f"Shifts without norm: {shifts_no_norm[:3]}")
print(f"Shifts with norm: {shifts_with_norm[:3]}")
print("✓ Covariate shift measurement working")

if __name__ == "__main__":
 print("Lab 4: Covariate Shift - PASSED")