Deep Learning

Deep learning is machine learning using neural networks with many stacked layers — "deep" literally refers to layer count, as opposed to the single or handful of layers used in classical neural networks from the 1980s and 90s. The core mathematics (stacked weighted sums, nonlinear activations, backpropagation to compute gradients) was worked out decades before deep learning became the dominant approach to AI; what changed wasn't the math, it was whether anyone could afford to actually run it at the depth and scale the math called for. At a high level, the shift from "neural networks exist" to "deep learning works" tracked three curves rising together:

AlexNet's 2012 ImageNet win is the single moment most historians point to as deep learning's actual beginning, and the reason is as much hardware as algorithm. AlexNet used 8 layers (shallow by today's standards) but was trained on two NVIDIA GTX 580 gaming GPUs instead of CPUs, cutting training time from what would have been months down to about a week, and it beat the next-best (non-deep-learning) approach's error rate by roughly 10 percentage points — a gap large enough that the entire computer vision field pivoted toward deep learning within about two years of that result. The algorithm existed before; the GPU made running it at a useful depth and dataset size actually affordable for the first time.

Every major deep learning architecture since AlexNet has been, in part, a response to what GPU hardware makes cheap versus expensive. Convolutional layers reuse the same small filter across an entire image specifically because that pattern maps efficiently onto a GPU's parallel cores; the transformer's self-attention mechanism, which replaced recurrent networks for most large-scale language modeling after 2017, won partly because attention is a large parallelizable matrix multiply while recurrent networks process one token at a time and can't be parallelized across the sequence — a direct consequence of what's fast on the hardware actually available, not a pure accuracy argument in isolation.

The scale that made deep learning famous (GPT, Stable Diffusion, AlphaFold) runs on purpose-built accelerator silicon, not general-purpose chips, for exactly the compute-density reasons that made AlexNet's GPUs matter in the first place. NVIDIA's Tensor Cores, Google's TPUs, and a growing field of custom AI ASICs all dedicate the overwhelming majority of their transistor budget to matrix-multiply-and-accumulate units rather than general logic, because a deep network's forward and backward passes are, at the hardware level, almost entirely large matrix multiplications — the same operation repeated at every layer, which is exactly the operation these chips are built to execute as densely as a process node allows.

Read deep learning's history as a story about chips catching up to a decades-old algorithm, not the algorithm catching up to a new idea: backpropagation and stacked nonlinear layers were both known in the 1980s, and the thirty-year gap before deep learning actually worked at scale is almost entirely explained by GPU compute, dataset size, and AI-specific silicon finally reaching the point where running that old math at real depth became economically possible.