neural network

A neural network is a machine-learning model that maps inputs to outputs by passing data through layers of simple units, each computing a weighted sum of its inputs followed by a nonlinear function. Loosely inspired by neurons in the brain, it is in practice a flexible function approximator: given enough data and parameters, it learns to turn pixels into labels, text into next-token probabilities, or audio into words — and it is the foundation of essentially all modern AI.\n\n```svg\n\n```\n\n**A single neuron is just a weighted sum plus a nonlinearity.** Each unit multiplies its inputs by a set of learned *weights*, adds a *bias*, and passes the result through an *activation function* such as ReLU. The weights and biases are the parameters the network learns; everything a trained model "knows" is encoded in their values. Stacking many such units into layers, and many layers into a network, is what gives the model its expressive power.\n\n**The nonlinearity is the whole point.** If each layer were purely a weighted sum, stacking layers would collapse into a single linear transformation — no matter how many you stacked, the network could only draw straight-line decision boundaries. The activation function breaks that linearity, letting successive layers compose into arbitrarily complex functions. This is why the choice of activation, and having any nonlinearity at all, is fundamental rather than a detail.\n\n**The forward pass turns input into a prediction.** Data enters at the input layer and flows forward, each layer transforming the representation from the one before, until the output layer produces a result — a class probability, a predicted value, a distribution over next tokens. Early layers tend to capture simple features and later layers combine them into abstract ones, a hierarchy the network discovers on its own rather than being told.\n\n**Learning happens by backpropagation and gradient descent.** A *loss function* measures how wrong the prediction is against the correct answer. Backpropagation applies the chain rule to compute how much each weight contributed to that error — the gradient — and gradient descent nudges every weight a small step in the direction that reduces the loss. Repeat over millions of examples and the weights settle into values that make good predictions. Training is nothing more than this loop run at enormous scale.\n\n**Architectures specialize the same idea.** A plain multilayer perceptron connects every unit to every unit in the next layer. Convolutional networks share weights across space and dominate vision; recurrent networks and, more recently, transformers handle sequences by mixing information across positions. They differ in how neurons are wired and which weights are shared, but underneath they are all the same machine: weighted sums, nonlinearities, and gradient-based learning.\n\n| Piece | What it does |\n|---|---|\n| Neuron / unit | computes a weighted sum of inputs, then a nonlinearity |\n| Weights & biases | the learned parameters that hold the model's knowledge |\n| Activation function | adds nonlinearity (ReLU, GELU, sigmoid) so layers can compose |\n| Loss function | measures prediction error to be minimized |\n| Backpropagation | computes gradients of the loss w.r.t. every weight |\n| Gradient descent | updates weights to reduce the loss, step by step |\n\nRead a neural network through a *learned-function-approximator* lens rather than a *brain* lens: the biological metaphor is where the name came from, but what the model actually is is a big parameterized function whose weights are tuned by gradient descent to fit data. Every architecture — MLP, CNN, transformer — is a different way of wiring weighted sums and nonlinearities, and every capability the model has comes not from mimicking neurons but from the optimization loop that adjusts those weights until the function does what the training data asks.\n