batch normalization layer normalization

**Normalization Techniques in Deep Learning** are **the family of methods that standardize activations within neural networks to stabilize training dynamics, enable higher learning rates, and reduce sensitivity to weight initialization — with Batch Normalization, Layer Normalization, Group Normalization, and Instance Normalization each normalizing along different dimensions for different use cases**. **Batch Normalization (BatchNorm):** - **Operation**: for each channel c, normalize activations across the batch dimension and spatial dimensions — μ_c and σ_c computed over (N, H, W) for each channel in a mini-batch; output = γ_c × (x - μ_c)/σ_c + β_c with learnable scale γ and shift β - **Training Behavior**: running mean and variance computed via exponential moving average during training — stored statistics used during inference for deterministic behavior independent of batch composition - **Benefits**: enables 10-30× higher learning rates, acts as regularizer (noise from mini-batch statistics), smooths the loss landscape — almost universally used in CNN architectures - **Limitations**: performance degrades with small batch sizes (< 16) due to noisy statistics; not applicable to variable-length sequences; batch-dependent behavior complicates distributed training and inference **Layer Normalization (LayerNorm):** - **Operation**: normalizes across all features within each sample independently — μ and σ computed over (C, H, W) for each sample; no dependence on batch dimension - **Use Cases**: standard in Transformer architectures (BERT, GPT, ViT) — batch-independent normalization essential for autoregressive models and variable-length sequence processing - **Pre-Norm vs. Post-Norm**: Pre-LayerNorm (normalize before attention/FFN) provides more stable training for deep Transformers — Post-LayerNorm (original Transformer) requires learning rate warmup but may achieve better final accuracy - **RMSNorm**: simplified variant using only root-mean-square normalization without centering — reduces computation by ~30% with comparable performance; used in LLaMA and other efficient Transformer architectures **Other Normalization Methods:** - **Group Normalization**: divides channels into G groups and normalizes within each group per sample — GroupNorm with G=32 achieves stable performance across all batch sizes; bridge between LayerNorm (G=1) and InstanceNorm (G=C) - **Instance Normalization**: normalizes each channel of each sample independently over spatial dimensions — standard for style transfer where per-channel statistics encode style information that should be normalized away - **Weight Normalization**: decouples weight vector magnitude from direction — reparameterizes W = g × v/||v|| with learned scalar g and unit direction v; more stable for RNNs than BatchNorm - **Spectral Normalization**: constrains the spectral norm (largest singular value) of weight matrices — stabilizes GAN discriminator training by limiting the Lipschitz constant **Normalization techniques are among the most impactful innovations in deep learning practice — choosing the right normalization method for the architecture and use case directly determines training stability, convergence speed, and final model quality.**