Weights

Weights are the numbers that multiply a neuron's inputs before they're summed — alongside bias terms, they are the actual learned content of a trained model, the part that starts as small random numbers and gets nudged, gradient step by gradient step, into values that encode whatever the model learned from its training data. Everything else in a neural network (the architecture, the activation functions, the layer connections) is fixed by the designer before training starts; weights are the one part that changes, which is why "training a model" and "learning the weights" mean exactly the same thing. At a high level, a weight's value moves through its lifecycle like this:

Weight initialization is not a formality — a bad starting point can make a network literally unable to train. If all weights start at exactly zero, every neuron in a layer computes the identical output and gets the identical gradient update forever, so the network can never break that symmetry and learn distinct features (the "zero-initialization trap"). If weights start too large, activations can explode layer over layer; too small, and they shrink toward zero before reaching the output — both versions of the vanishing/exploding problem that plagued early deep networks. Xavier/Glorot and He initialization fix this by scaling the random starting values specifically to the number of inputs and outputs a layer has, keeping the signal's variance roughly stable as it passes through dozens or hundreds of layers.

A weight's numeric precision — not just its value — is now one of the biggest cost levers in production AI. Weights trained at 32-bit or 16-bit floating point can be converted to 8-bit or 4-bit integers after training (post-training quantization) with only small accuracy loss, roughly halving or quartering the memory a model needs; NVIDIA's Hopper and Blackwell architectures added dedicated FP8 and FP4 tensor-core support specifically because the industry standardized on quantized weights for production inference, not as an edge-case optimization but as the default deployment path for models serving real traffic.

Weight sharing is the other major lever, and it works by making one weight do the job of many. A convolutional layer's filter weights are reused at every spatial position in an image rather than learning separate weights per pixel location — this single design choice is most of why CNNs need vastly fewer parameters than an equivalent fully connected network would. Model merging and weight averaging take this further: combining the weights of multiple independently fine-tuned models by literally averaging their values, a cheap technique that often produces a model more robust than any single one of its ingredients, with zero additional training.

Read a weight as the atomic unit of everything a trained model knows: its starting value decides whether training even converges, its final value is the actual compressed knowledge extracted from the training data, and its stored numeric precision is a direct line item against the HBM capacity and GPU memory bandwidth a deployment needs — the smallest number in the system, carrying the largest share of the engineering tradeoffs.