Parameters

Parameters are the individual numeric values — weights and biases — inside a model that get adjusted during training and then stay fixed during inference; the parameter count quoted for a model (7 billion, 70 billion, 405 billion) is literally a count of these numbers, not a measure of the model's intelligence or code complexity. Every parameter is a single scalar a GPU has to store and touch during every matrix multiply, which is why parameter count is the number that converts most directly into memory footprint, training cost, and inference latency rather than staying an abstract architectural detail. At a high level, a parameter's lifecycle inside a training run looks like this:

Parameter count sets a hard memory floor before a single computation runs. A model stored at FP16 precision needs 2 bytes per parameter just for the weights, plus roughly the same again for the optimizer state (momentum and variance terms) during training — so training a 70-billion-parameter model from scratch needs on the order of 1.4TB combined for weights, gradients, and optimizer state with a standard Adam optimizer, before any activation memory for the actual data passing through. That number is why training runs are split across racks of NVIDIA H100 or B200 GPUs using techniques like ZeRO and fully sharded data parallelism — a single 80GB-HBM GPU can't hold that much state on its own.

Not all parameters do the same job, and some are cheaper than others to compute with. A dense model like early GPT-3 activates every parameter for every token processed; a mixture-of-experts model like Mixtral 8x7B has roughly 47 billion total parameters but only routes each token through about 13 billion of them — a deliberate split between total parameter count (knowledge capacity, all of which still has to sit in HBM) and active parameter count (the actual FLOPs spent per token). Parameter-efficient fine-tuning methods like LoRA exploit the same distinction from the other direction: instead of updating all 70 billion original parameters, LoRA freezes them and trains a tiny set of new low-rank matrices (often under 1% of the original parameter count), cutting fine-tuning memory and compute by orders of magnitude while keeping nearly all of the original model's capability intact.

Scaling laws turned parameter count from a design choice into a predictable science. DeepMind's Chinchilla paper showed that many earlier large models were actually over-parameterized relative to how much training data they'd seen, and that for a fixed compute budget, a smaller model trained on proportionally more tokens often beats a larger model undertrained on too little data — which is why frontier labs now jointly tune parameter count and token count together rather than just maximizing parameters, since more parameters with insufficient data is wasted HBM and wasted GPU-hours, not free capability.

Read parameter count as a direct line item against a data center's GPU, HBM, and power budget, not as a marketing number: every billion parameters announced is a commitment to bytes of memory, interconnect bandwidth, and energy that has to physically exist on silicon before the model answers a single query — the same compounding-cost discipline that decides whether a chip's 60-100+ layers yield a working die also decides, one level up, whether a given parameter count is actually deployable at all.