Normalization layers are the quiet workhorses that make deep networks trainable at all. Left alone, the activations flowing through a deep stack drift in scale and distribution from layer to layer, so gradients explode or vanish and the optimizer stalls. A normalization layer re-centers and re-scales those activations back to a well-behaved range at every step, which smooths the loss landscape, lets you use a much higher learning rate, and makes training far less sensitive to weight initialization. The whole transformer era rests on getting this one detail right.\n\nBatch normalization normalizes each feature across the batch dimension. For a given channel it computes the mean and variance over all the examples in the mini-batch, standardizes, then applies a learnable scale and shift. It was the breakthrough that made very deep CNNs trainable, but it has two awkward properties: it needs a reasonably large batch to estimate stable statistics, and it behaves differently at training time (batch statistics) than at inference (running averages), which makes it a poor fit for sequence models and small-batch or variable-length workloads.\n\nLayer normalization normalizes across the feature dimension instead, one token at a time. Because it computes statistics within a single example, it is completely independent of batch size and behaves identically in training and inference. That batch-independence is exactly what recurrent and Transformer architectures need, which is why LayerNorm — not BatchNorm — is the default inside every attention block.\n\nRMSNorm strips LayerNorm down to just the scaling term. It drops the mean-subtraction step and rescales purely by the root-mean-square of the activations, with a single learnable gain and no bias. It costs less compute and memory while matching LayerNorm's quality in practice, which is why modern large models such as the LLaMA family and many others adopt it as the default. GroupNorm sits between BatchNorm and LayerNorm by normalizing over groups of channels, and is common in vision models where batches are small.\n\nWhere you place the normalization matters as much as which one you pick. The original Transformer used post-norm (normalize after the residual add), which is expressive but needs careful learning-rate warmup and can be unstable at depth. Nearly every modern large model instead uses pre-norm (normalize inside the residual branch, before each sublayer), which keeps a clean gradient path through the residual stream and trains stably to hundreds of layers. The learnable gain and bias parameters mean a normalization layer can always undo its own normalization if the network needs to, so it never costs the model representational power.\n\n| Norm | Reduces over | Batch-dependent? | Train == inference? | Typical home |\n|---|---|---|---|---|\n| BatchNorm | Batch (per channel) | Yes | No (running stats) | CNNs, large batches |\n| LayerNorm | Features (per token) | No | Yes | Transformers, RNNs |\n| RMSNorm | Features, no mean | No | Yes | Modern LLMs (LLaMA-style) |\n| GroupNorm | Channel groups | No | Yes | Vision, small batches |\n\n``svg\n (x - mean) / sqrt(var + e) + b\n RMSNorm: y = g x / sqrt(mean(x^2) + e)\n RMSNorm drops the mean-centering and the bias -> cheaper\n\n \n 2 - Where the norm sits in the block\n\n \n Post-norm (original)\n \n Sublayer (attn / FFN)\n + residual\n \n Normalize\n needs warmup; unstable when deep\n\n \n Pre-norm (modern)\n \n Normalize\n \n Sublayer (attn / FFN)\n + residual\n clean gradient path; trains to 100s of layers\n\n \n \n Why it works\n Normalization smooths the loss landscape, so gradients stay well-scaled and you can crank the learning rate.\n The learnable gain/bias let the layer undo the normalization if needed, so it never costs representational power.\n \n Transformers -> LayerNorm / RMSNorm, pre-norm placement\n \n CNNs / large batch -> BatchNorm\n\n``\n\nThe temptation is to think of normalization as a preprocessing nicety — something you sprinkle in because a paper did. It is better read as optimization infrastructure: the layer that keeps the activation distribution conditioned so the optimizer sees a smooth, well-scaled loss surface at every depth. Which variant you reach for, and where you place it, is a statement about how you want gradients to flow. Read normalization through a conditioning-the-optimization lens rather than a fixing-covariate-shift lens, and the choice between BatchNorm, LayerNorm, and RMSNorm — and between pre-norm and post-norm — stops being folklore and becomes a direct consequence of your batch structure and your network depth.