backpropagation

**Backpropagation** is the algorithm that lets a neural network learn. After the network makes a prediction and we measure how wrong it was, backpropagation efficiently computes how much each of the millions or billions of weights contributed to that error — the gradient — so an optimizer can nudge every weight in the direction that reduces the loss. It is, at heart, the chain rule from calculus applied systematically across a computation graph, and it is what makes training deep networks tractable at all. The diagram shows the two passes: forward to get the error, backward to distribute the blame.\n\n```svg\n\n```\n\n**The forward pass computes the prediction and the loss.** Input data flows layer by layer through the network — each layer multiplying by its weights and applying a nonlinearity — until it produces an output. That output is compared to the correct answer with a loss function, giving a single number that measures how wrong the network currently is. Along the way, each layer stores the intermediate values it computed, because the backward pass will need them.\n\n**The backward pass applies the chain rule in reverse.** Starting from the loss, backpropagation works backward through the layers, computing at each step how the loss changes with respect to that layer's inputs and weights. The key efficiency is reuse: the gradient at layer *k* is built directly from the gradient already computed at layer *k+1*, multiplied by a local derivative. Nothing is recomputed, which is why a full gradient over billions of parameters costs only about twice a forward pass.\n\n**Gradients are just directions for improvement.** The gradient with respect to a weight answers one question — if I increase this weight slightly, does the loss go up or down, and how fast? Backpropagation produces that answer for every weight at once. It does not change anything itself; it only measures. The actual learning step is handed to an optimizer such as SGD or Adam.\n\n**The vanishing-gradient problem shaped modern architectures.** When gradients are repeatedly multiplied through many layers, they can shrink toward zero (or blow up), stalling learning in the earliest layers. Much of deep-learning design — ReLU activations, residual/skip connections, careful normalization and initialization — exists specifically to keep gradients healthy as they propagate back through great depth.\n\n**It requires stored activations, which is why training is memory-hungry.** Because the backward pass needs the intermediate values from the forward pass, they must be kept in memory until used. This is a major reason training a model costs far more memory than running it, and it motivates techniques like gradient (activation) checkpointing, which trade recomputation for reduced memory.\n\n| Step | Direction | Produces | Cost |\n|---|---|---|---|\n| Forward pass | input → output | prediction + loss | one pass |\n| Backward pass | loss → inputs | gradient for every weight | about one pass |\n| Optimizer step | — | updated weights | cheap |\n| Repeat | over many batches | a trained model | the whole training run |\n\nRead backpropagation through a *credit-assignment* lens rather than a *magic-learning* lens: the entire algorithm is a bookkeeping method for answering "how much did each weight contribute to this mistake?" without redoing work, by caching local derivatives on the way in and multiplying them together on the way out. Every scaling and stability trick in deep learning — residual connections, normalization, mixed precision, activation checkpointing — is ultimately about keeping that backward flow of credit accurate, fast, and affordable.\n