Weight initialization is the choice of what values a network's parameters hold before the first gradient step — and it is far less innocent than it sounds. Set the initial random weights badly and a deep network never trains at all: the signal either fades to nothing or blows up to infinity as it passes through the layers, and the gradients do the same on the way back. The reason Xavier and He initialization exist, and the reason they are calculated from the number of connections into and out of each layer rather than pulled from a fixed range, is a single governing goal — keep the variance of the activations and gradients roughly constant as they propagate through a deep stack, so that signal survives the trip in both directions.\n\nThe core problem is variance that compounds layer by layer. Each layer multiplies its input by a weight matrix and sums, and that sum's variance depends on how many inputs feed it (the fan-in) and how large the weights are. Chain many layers together and the effect is multiplicative: if each layer shrinks the variance even slightly, activations decay geometrically toward zero over dozens of layers (vanishing), and if each layer amplifies it, they explode toward infinity (exploding). Both are fatal — a vanished signal carries no information and produces vanishing gradients that stall learning, while an exploded one produces NaNs. Good initialization is the requirement that, on average, each layer neither shrinks nor grows the variance, so a unit-scale input stays unit-scale a hundred layers deep.\n\nXavier (Glorot) initialization solves this for symmetric activations by balancing fan-in and fan-out. Derived assuming an activation that is roughly linear around zero — like tanh or sigmoid — Xavier sets the weight variance to 2 / (fan_in + fan_out), a compromise that keeps activation variance stable on the forward pass and gradient variance stable on the backward pass. Sampling weights from a normal or uniform distribution scaled this way was the first principled recipe that let deep networks train reliably, replacing the ad-hoc "small random numbers" that had quietly capped network depth for years.\n\nHe (Kaiming) initialization corrects Xavier for ReLU, which throws away half the signal. ReLU sets all negative activations to zero, so on average it halves the variance passing through — a factor Xavier's derivation did not account for. He initialization compensates by doubling the scale, setting the weight variance to 2 / fan_in, which restores the balance for ReLU and its relatives (GELU, etc.). This is why modern convolutional and feedforward networks default to He, while Xavier lingers where tanh/sigmoid are used. In today's very deep transformers the story is softened but not erased: normalization layers (BatchNorm, LayerNorm) and residual connections absorb much of the sensitivity to initial scale, and large models add tricks like scaling residual branches down by the number of layers — but they still start from a carefully chosen small-variance init, because even normalized residual networks train better when the signal starts at the right scale.\n\n| Scheme | Weight variance | Designed for |\n|---|---|---|\n| "Small random" (naïve) | Fixed small range | Nothing — caps depth |\n| Xavier / Glorot | 2 / (fan_in + fan_out) | tanh, sigmoid (symmetric) |\n| He / Kaiming | 2 / fan_in | ReLU, GELU (half-rectified) |\n| Orthogonal | Norm-preserving matrix | RNNs, very deep nets |\n| + Norm & residuals | Reduce init sensitivity | Modern transformers |\n\n``svg\n\n``\n\nThe unhelpful way to think about weight initialization is as a throwaway detail — just fill the matrices with small random numbers and let training sort it out. The useful way is to see it as setting the scale of the signal at the entrance to a deep pipeline, where every layer multiplies what came before, so a scale that is even slightly off compounds into vanishing or exploding activations and gradients before learning can begin. Xavier keeps the variance balanced for symmetric activations by averaging fan-in and fan-out; He corrects for the half of the signal that ReLU discards by doubling the scale over fan-in; normalization and residuals later make deep networks more forgiving but never make the starting scale irrelevant. Read weight initialization through a keep-the-signal-variance-alive lens rather than a just-pick-small-random-numbers lens, and the specific formulas stop looking arbitrary and become exactly what they are — the unique scales that let a signal cross a hundred layers without dying or diverging.