model training

Training is the process of teaching a machine-learning model by repeatedly showing it data and adjusting its internal parameters — its weights — until its predictions match the desired answers. It is where a model's capabilities actually come from: an untrained network is random, and everything it eventually "knows" is written into billions of weights by this optimization loop. Training a frontier model is also the single most expensive thing in modern AI, which is why so much engineering goes into doing it efficiently.\n\n```svg\n\n```\n\n**Training is a loop, repeated on batch after batch.** A batch of examples is fed forward through the model to produce predictions; a loss function scores how wrong those predictions are against the correct labels; backpropagation computes how each weight contributed to that error; and an optimizer nudges every weight a small step in the direction that reduces the loss. Run this loop over enough data for enough steps and the weights converge toward values that make good predictions. Nothing more mysterious is happening — learning is this cycle at enormous scale.\n\n**The loss function defines what "good" means.** The whole procedure only optimizes the objective you write down, so the loss is the model's true goal. Cross-entropy for classification and next-token prediction, mean-squared error for regression, and more elaborate objectives for alignment all steer the weights differently. A model does exactly what its loss rewards, which is why choosing and shaping the loss is one of the most consequential decisions in training.\n\n**Backpropagation and the optimizer are the learning mechanism.** Backprop applies the chain rule to get the gradient — the sensitivity of the loss to each weight — in a single backward sweep. The optimizer then takes the step: plain SGD moves against the gradient, while Adam and its variants adapt the step size per parameter using running estimates of the gradient. A *learning-rate schedule* controls how big those steps are over the course of training, typically warming up and then decaying, because too large a step diverges and too small a step crawls.\n\n**Training is much heavier than inference, and that is inherent.** A forward pass alone — what inference does — is comparatively cheap. Training must also store every layer's activations so backprop can use them, run the backward pass, and hold optimizer state, so its memory and compute cost is several times higher per token. This asymmetry is why training runs on large GPU or TPU clusters for weeks while the same model can later serve requests far more cheaply.\n\n**Scale is spread across many devices.** Frontier models do not fit on one accelerator, so training is distributed: *data parallelism* replicates the model and splits the batch, *tensor* and *pipeline parallelism* split the model itself across devices, and gradients are synchronized every step with collective operations like all-reduce. At this scale, interconnect bandwidth and keeping thousands of chips busy — not raw arithmetic — usually set the wall-clock training time.\n\n**Generalization, not memorization, is the goal.** Success is measured on data the model has never seen. Techniques like weight decay, dropout, data augmentation, and early stopping fight *overfitting*, where a model memorizes its training set but fails to generalize. Held-out validation and test sets are how practitioners tell learning apart from memorization.\n\n| Stage | What happens | Key choices |\n|---|---|---|\n| Forward pass | compute predictions from a batch | model architecture, batch size |\n| Loss | score error vs labels | objective (cross-entropy, MSE, ...) |\n| Backpropagation | gradient of loss per weight | automatic differentiation |\n| Optimizer step | update weights | SGD vs Adam, learning-rate schedule |\n| Regularization | keep the model general | weight decay, dropout, early stopping |\n\n| | Training | Inference |\n|---|---|---|\n| Passes | forward + backward | forward only |\n| Memory | stores activations + optimizer state | weights + KV cache |\n| Cost | very high, done once | lower, paid per request |\n| Goal | fit weights to data | apply fixed weights |\n\nRead training through an *optimization-loop* lens rather than a *magic* lens: a model learns because a loss function defines what wrong means, backpropagation measures how each weight contributes to being wrong, and an optimizer repeatedly nudges the weights to be a little less wrong. Every technique in the field — better losses, smarter optimizers, learning-rate schedules, distributed parallelism, regularization — is a refinement of that one loop, aimed at making it converge faster, scale across more chips, or generalize better to data the model has never seen.\n