Batch Normalization Layer Normalization Group Norm
# Batch Normalization, Layer Normalization & Group Norm
## Introduction & Motivation
Normalization techniques: stabilize training. Batch normalization: normalize by batch statistics. Layer normalization: normalize per token; sequence independent. Group normalization: intermediate approach. Applications: CNNs, transformers, RNNs, stabilizing deep networks.
Motivation: Internal covariate shift; normalization reduces. Enables higher learning rates; better generalization.
Applications: Deep networks, CNNs, transformers.
---
## Core Concepts & Theory
### Batch Normalization
Normalize activations per batch; learnable scale/shift.
### Layer Normalization
Normalize per token/feature; batch independent.
### Group Normalization
Partition channels; normalize per group.
---
## Mathematical Formulation
Batch normalization:
$$\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} \quad y_i = \gamma \hat{x}_i + \beta$$
where μ_B = batch mean, σ_B = batch std.
Layer normalization:
$$\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \quad y = \gamma \hat{x} + \beta$$
where normalization per sample/position.
Group normalization:
$$ ext{norm per channel group; } N_g ext{ groups}$$
---
## Advanced Theory & Extensions
### Batch Renormalization
Address train-test mismatch; reparameterization.
### Weight Normalization
Decouple magnitude from direction.
### Instance Normalization
Normalize per sample per channel; style transfer.
---
## Computational Considerations
Batch norm: O(N·d) forward, O(N·d) backward.
Layer norm: O(d) per sample; no batch dependency.
Group norm: O(N·d/G) where G = groups.
---
## Practical Implementation Strategies
### Momentum Update
Running mean/variance; exponential moving average.
### Epsilon Value
Numerical stability; typical 1e-5.
### Affine Parameters
Scale/shift learnable; usually enabled.
---
## Benchmark Datasets & Evaluation
ImageNet: BatchNorm standard; CNN baseline.
CIFAR-10: LayerNorm competitive on vision.
NLP Benchmarks: LayerNorm dominant; transformer standard.
---
## Key Challenges & Limitations
### Train-Test Mismatch
Batch stats change; running stats used at test.
### Small Batch Issues
BatchNorm unstable; LayerNorm robust.
### Synchronized BatchNorm
Distributed training; synchronize across GPUs.
---
## Hyperparameter Tuning
BatchNorm momentum: 0.1-0.99; exponential average.
Epsilon: 1e-5 typical; numerical stability.
LayerNorm epsilon: 1e-12 standard; shape preservation.
---
## Real-World Applications & Case Studies
ImageNet: ResNet + BatchNorm; accuracy plateau breakthrough.
Transformers: LayerNorm standard; Pre-LN variants.
Distributed Training: SyncBN; accuracy consistency.
---
## Integration with Other Methods
Normalization + Regularization → complementary.
Normalization + Skip Connections → gradient flow.
---
## Summary & Key Takeaways
Normalization techniques via batch, layer, and group statistics stabilize training and improve generalization.
Principles:
1. BatchNorm: batch-dependent normalization.
2. LayerNorm: batch-independent; feature-wise.
3. GroupNorm: intermediate; group-wise.
4. Affine: learnable scale/shift.
5. Momentum: running statistics.
---
---
## Appendix: Practical Labs
### Lab 1: Batch Normalization
import numpy as np
class BatchNorm:
def __init__(self, num_features, momentum=0.9, epsilon=1e-5):
self.momentum = momentum
self.epsilon = epsilon
self.gamma = np.ones(num_features)
self.beta = np.zeros(num_features)
self.running_mean = np.zeros(num_features)
self.running_var = np.ones(num_features)
def forward(self, x, training=True):
"""Batch normalization forward"""
if training:
batch_mean = x.mean(axis=0)
batch_var = x.var(axis=0)
# Update running stats
self.running_mean = self.momentum * self.running_mean + (1 - self.momentum) * batch_mean
self.running_var = self.momentum * self.running_var + (1 - self.momentum) * batch_var
else:
batch_mean = self.running_mean
batch_var = self.running_var
# Normalize
x_norm = (x - batch_mean) / np.sqrt(batch_var + self.epsilon)
# Scale and shift
y = self.gamma * x_norm + self.beta
return y
# Test
np.random.seed(42)
bn = BatchNorm(5)
x = np.random.randn(32, 5)
y = bn.forward(x, training=True)
assert y.shape == x.shape, "Output shape"
assert not np.allclose(y.mean(axis=0), 0, atol=0.1), "Normalized"
print("✓ Batch normalization working")
if __name__ == "__main__":
print("Lab 1: BatchNorm - PASSED")### Lab 2: Layer Normalization
import numpy as np
class LayerNorm:
def __init__(self, normalized_shape, epsilon=1e-12):
self.normalized_shape = normalized_shape
self.epsilon = epsilon
self.gamma = np.ones(normalized_shape)
self.beta = np.zeros(normalized_shape)
def forward(self, x):
"""Layer normalization forward"""
# Compute mean and variance per sample
mean = x.mean(axis=-1, keepdims=True)
var = x.var(axis=-1, keepdims=True)
# Normalize
x_norm = (x - mean) / np.sqrt(var + self.epsilon)
# Scale and shift
y = self.gamma * x_norm + self.beta
return y
# Test
np.random.seed(42)
ln = LayerNorm(5)
x = np.random.randn(32, 5)
y = ln.forward(x)
assert y.shape == x.shape, "Output shape"
assert np.allclose(y.mean(axis=-1), 0, atol=0.1), "Per-sample normalized"
print("✓ Layer normalization working")
if __name__ == "__main__":
print("Lab 2: LayerNorm - PASSED")### Lab 3: Group Normalization
import numpy as np
class GroupNorm:
def __init__(self, num_channels, num_groups, epsilon=1e-5):
self.num_channels = num_channels
self.num_groups = num_groups
self.epsilon = epsilon
self.gamma = np.ones(num_channels)
self.beta = np.zeros(num_channels)
def forward(self, x):
"""Group normalization forward"""
N, C, H, W = x.shape
# Reshape to groups
x_grouped = x.reshape(N, self.num_groups, C // self.num_groups, H, W)
# Normalize per group
mean = x_grouped.mean(axis=(2, 3, 4), keepdims=True)
var = x_grouped.var(axis=(2, 3, 4), keepdims=True)
x_norm = (x_grouped - mean) / np.sqrt(var + self.epsilon)
# Reshape back
x_norm = x_norm.reshape(N, C, H, W)
# Scale and shift
y = self.gamma[np.newaxis, :, np.newaxis, np.newaxis] * x_norm + self.beta[np.newaxis, :, np.newaxis, np.newaxis]
return y
# Test
np.random.seed(42)
gn = GroupNorm(8, num_groups=2)
x = np.random.randn(2, 8, 4, 4)
y = gn.forward(x)
assert y.shape == x.shape, "Output shape"
print("✓ Group normalization working")
if __name__ == "__main__":
print("Lab 3: GroupNorm - PASSED")### Lab 4: Normalization Comparison
import numpy as np
def compare_normalizations(x):
"""Compare mean/var after different normalizations"""
# Batch norm
batch_mean = x.mean(axis=0)
batch_var = x.var(axis=0)
# Layer norm
layer_mean = x.mean(axis=-1, keepdims=True)
layer_var = x.var(axis=-1, keepdims=True)
# Instance norm
inst_mean = x.mean()
inst_var = x.var()
return {
"batch_mean_std": batch_mean.std(),
"batch_var_std": batch_var.std(),
"layer_mean_std": layer_mean.std(),
"layer_var_std": layer_var.std(),
"instance_mean": inst_mean,
"instance_var": inst_var
}
# Test
np.random.seed(42)
x = np.random.randn(32, 128)
stats = compare_normalizations(x)
assert all(np.isfinite(v) for v in stats.values()), "All stats finite"
print("✓ Normalization comparison working")
if __name__ == "__main__":
print("Lab 4: Comparison - PASSED")