classic architectures

The history of deep learning is really a history of *inductive bias* — the structural assumption each architecture bakes into its wiring about what kind of pattern matters. A fully-connected network assumes nothing and must learn everything from data; a convolutional network assumes that nearby pixels relate and that a cat is a cat wherever it appears; a recurrent network assumes that order and recency matter; a transformer assumes that any element might relate to any other. Every classic architecture is best understood not as a bag of tricks but as one such assumption made concrete, and the field's progression is the story of finding the right bias for each kind of data.\n\n**The multilayer perceptron (MLP) is the primitive with no structural assumption at all.** Every input connects to every neuron, stacked into layers with a nonlinearity between them. It is a universal function approximator, but that generality is also its weakness: with no built-in notion of space, time, or relation, it needs enormous data to learn structure that other architectures get for free. The MLP never disappeared, though — it lives on as the building block *inside* larger models, most visibly as the feed-forward block in every transformer layer.\n\n**Convolutional networks (CNNs) bake in locality and translation invariance, and they owned computer vision for a decade.** By sliding small shared filters across an image, a CNN assumes that useful features are local and appear the same wherever they occur, which slashes the parameter count and matches the structure of natural images. From AlexNet in 2012 through the ResNet era, CNNs defined vision — until Vision Transformers showed that with enough data, attention could match or beat them.\n\n**Recurrent networks (RNNs) and their gated descendant the LSTM assume sequential, temporal structure.** They process a sequence one element at a time, carrying a hidden state that summarizes everything seen so far. This made them the default for language and speech before 2017, but their step-by-step nature is hard to parallelize and their memory of distant context is weak. The transformer replaced recurrence with attention — looking at all positions at once — which parallelizes cleanly and models long-range relationships directly, and is now the dominant architecture across text, vision, and audio.\n\n**The generative families — GANs, VAEs, and diffusion models — are organized by objective rather than by layer type.** A GAN pits a generator against a discriminator in a minimax game; a VAE learns a probabilistic latent space through an encoder-decoder; a diffusion model learns to reverse a noising process. They can be built from convolutional or transformer backbones, and what unites them is a way of *framing the learning problem* to produce new data rather than to classify existing data. Threading through all of these is one enabling trick — the residual (skip) connection introduced by ResNet — which lets gradients flow through very deep stacks and is what made the deep versions of every one of these architectures trainable.\n\n| Family | Inductive bias | Native data | Signature operation | Status today |\n|---|---|---|---|---|\n| MLP | None (fully connected) | Vectors, tabular | Dense matmul | Building block (FFN) |\n| CNN | Locality + translation invariance | Images, grids | Convolution | Vision (largely ceded to ViT) |\n| RNN / LSTM | Sequential / temporal | Sequences | Recurrence + gates | Legacy sequence models |\n| Transformer | Any-to-any relation | Anything tokenizable | Self-attention | Dominant everywhere |\n| GAN / VAE / Diffusion | Generative objective | Images, audio, text | Adversarial / latent / denoise | Standard for generation |\n\n```svg\n\n```\n\nThe unhelpful way to learn these architectures is as a chronological list of models to memorize. The useful way is to see them as a small set of answers to a single question: what does this network get to assume for free, and what must it learn from scratch? A CNN gets locality for free and pays for it when the data is not grid-shaped; an RNN gets sequence for free and pays for it in parallelism; a transformer assumes almost nothing about structure and pays for it in data and compute, which is exactly why it generalizes across so many domains. Read an architecture through an inductive-bias lens rather than a which-model-is-newest lens, and the whole family tree — MLP to CNN to RNN to transformer, and the generative variants hanging off each — organizes itself around what assumption fits the problem in front of you.