transformer

The **Transformer** is the neural-network architecture introduced in the 2017 paper *Attention Is All You Need*, and it is the foundation of virtually every modern large language model, image generator, and speech system. Its breakthrough was replacing the sequential, step-by-step processing of earlier recurrent networks with a mechanism — self-attention — that looks at an entire sequence at once and lets every element directly consult every other. That single change made it possible to train on far more data, in parallel, than anything before it. The diagram shows the repeating block that gets stacked to build the whole model.\n\n```svg\n\n```\n\n**Self-attention is the core idea.** For every token, the model produces three vectors — a query, a key, and a value. It compares each token's query against all the keys to decide how much attention to pay to every other token, normalizes those scores with a softmax, and returns a weighted blend of the values. The result is a new representation of each token that has absorbed exactly the context it needs, whether the relevant word is one position away or a thousand.\n\n**Multi-head attention looks in several ways at once.** Rather than a single attention computation, the block runs several in parallel — different "heads" that can specialize, one tracking syntax, another coreference, another local phrasing. Their outputs are concatenated and projected back together, giving the model multiple relationship types per layer.\n\n**The feed-forward network processes each token alone.** After attention has mixed information across positions, a small two-layer network is applied independently to every token: expand to a wider dimension, apply a nonlinearity, project back. This is where much of the model's raw capacity and stored knowledge lives. Attention decides *what to combine*; the feed-forward layer decides *what to do with it*.\n\n**Residual connections and normalization make depth trainable.** Each sub-layer's output is added back to its input (a residual, or skip, connection) and normalized. This keeps gradients flowing cleanly through dozens or hundreds of stacked layers, which is what lets Transformers go deep without the signal degrading.\n\n**Parallelism is the reason it won.** Because there is no recurrence, all positions in a sequence are processed simultaneously during training — a perfect match for the wide, parallel arithmetic of GPUs and TPUs. Recurrent networks had to march through a sequence one step at a time; the Transformer turned language modeling into big matrix multiplications, and that is exactly what modern accelerators do fastest.\n\n| Piece | What it does | Question it answers |\n|---|---|---|\n| Query / Key / Value | per-token vectors for attention | what am I looking for, offering, carrying |\n| Attention scores | Q·Kᵀ scaled, then softmax | which tokens matter to me |\n| Multi-head | parallel attention subspaces | what relationships exist at once |\n| Feed-forward | per-token transformation | what to make of the mixed context |\n| Residual + norm | add input back, normalize | how to stay trainable when deep |\n\nRead a Transformer through an *all-at-once attention* lens rather than a *sequence-processing* lens: earlier models understood a sentence by walking through it word by word, carrying a running memory, while the Transformer lays the whole sequence out and lets every token pull directly from every other in a single parallel step. That shift is why it trains efficiently at massive scale, why context length is such a central design axis, and why "attention" — not recurrence or convolution — became the organizing principle of modern AI.\n