Neuron

A neuron, in a neural network, is the basic computational unit: it takes a set of numeric inputs, multiplies each by a learned weight, sums them together with a bias term, and passes that sum through a nonlinear activation function to produce a single output number. The name borrows loosely from biological neurons, but the artificial version is dramatically simpler — no spiking, no chemistry, just a weighted sum and a nonlinearity — and that simplicity is exactly what makes it cheap enough to replicate by the billions on a GPU. At a high level, a single neuron's computation looks like this:

A single neuron can only draw a straight line; stacking millions of them is what lets a network learn curves. Without a nonlinear activation function, stacking layers of neurons would collapse mathematically into one giant linear equation, no matter how many layers were added — a network built entirely of linear operations can only separate data with a straight line or flat plane, which is useless for almost every real task (image recognition, language, anything with real-world curvature in its decision boundary). The nonlinearity inside each neuron is what lets a deep stack of them approximate arbitrarily complex functions — this is the literal mathematical content behind "universal approximation," the property that makes deep learning work at all.

Activation function choice is a real engineering decision with real training consequences, not a minor stylistic detail. Early networks used sigmoid and tanh, but both saturate — their output flattens near 0 or 1 for large inputs, which drives their gradient toward zero and stalls learning in deep networks (the vanishing-gradient problem). ReLU (simply outputting zero for negative inputs and the input itself for positive ones) fixed most of this and became the default for over a decade because it's nearly free to compute and keeps gradients flowing; GELU and SiLU, smoother variants used in most modern transformers including GPT-class models, trade a small amount of extra compute for slightly better training dynamics at scale.

A neuron's weighted sum is literally a dot product, which is exactly the operation GPUs and AI accelerators are built to do at enormous scale. A layer with 12,288 neurons, each connected to 12,288 inputs, is a single 12,288×12,288 matrix multiply — this is why NVIDIA's Tensor Cores, Google's TPUs, and every modern AI chip dedicate the overwhelming majority of their silicon area specifically to matrix-multiply-and-accumulate units rather than general-purpose logic: the fundamental math of a neuron, repeated by the billions, is what a chip is actually being designed around.

Read a neuron not as a brain cell analogy but as the smallest repeatable unit of a much larger matrix operation: a layer is a matrix multiply, a network is a sequence of matrix multiplies and nonlinearities, and the entire point of modern AI silicon — from NVIDIA's Tensor Cores to custom accelerator ASICs — is executing that one simple weighted-sum-and-activation pattern as many trillions of times per second as the transistor budget allows.