transformer architecture
**Transformer architecture is the neural network design that powers virtually every modern large language model, including GPT, Claude, Gemini, and Llama.** Introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al. at Google, the transformer replaced recurrent and convolutional sequence models with a mechanism called self-attention that processes all positions in a sequence simultaneously. This parallelism made transformers dramatically faster to train on modern accelerators and enabled scaling to hundreds of billions of parameters. The architecture has since expanded beyond language into vision (ViT), protein folding (AlphaFold), code generation, speech, music, robotics, and scientific discovery, making it arguably the most consequential neural network design in the history of deep learning.
**Self-attention is the core mechanism that gives the transformer its power.** For each token in a sequence, the model computes three vectors — query (Q), key (K), and value (V) — by multiplying the token embedding by learned weight matrices. Attention scores are calculated as the scaled dot product of queries and keys, then passed through a softmax to produce weights that determine how much each token attends to every other token. The output is a weighted sum of the value vectors. Multi-head attention runs this process in parallel across multiple subspaces (typically 8 to 128 heads), allowing the model to capture different types of relationships simultaneously — syntactic structure in one head, semantic similarity in another, positional patterns in a third. The attention computation for a single head is
$$\text{Attention}(Q,K,V) = \text{softmax}\!\Bigl(\frac{QK^T}{\sqrt{d_k}}\Bigr)V$$
where d_k is the dimension of each key vector. The square-root scaling prevents dot products from growing too large in high dimensions, which would push softmax into regions with vanishingly small gradients. This operation has quadratic complexity in sequence length, which is why context-length extension has become a major research focus.
**Each transformer block combines attention with a feed-forward network in a residual structure.** After multi-head attention, the output is added back to the input (residual connection) and passed through layer normalization, which stabilizes training by normalizing activations across the feature dimension. The normalized output then passes through a position-wise feed-forward network — typically two linear transformations with a GeLU or SiLU activation between them — followed by another residual connection and layer normalization. This attention-then-FFN pattern repeats for every layer in the model. Modern architectures stack 32 to 128 such blocks. The feed-forward network is where the model stores factual knowledge and performs pattern transformation, while attention handles routing information between positions.
**Positional encoding solves the ordering problem created by attention's permutation invariance.** Because self-attention treats its input as a set rather than a sequence, the model needs explicit position information. The original transformer used fixed sinusoidal encodings at different frequencies for each dimension, but modern architectures have moved to learned position embeddings or rotary position embeddings (RoPE). RoPE encodes relative position by rotating query and key vectors in pairs of dimensions, and has become the dominant approach in models like Llama, Mistral, and Qwen because it generalizes better to sequence lengths beyond training and integrates naturally with attention computation.
**The original transformer used an encoder-decoder structure, but modern variants have specialized.** The encoder processes input bidirectionally — each token attends to all others — making it ideal for understanding tasks. The decoder generates output autoregressively, using causal masking so each token can only attend to previous positions. Encoder-only models like BERT excel at classification, extraction, and retrieval. Decoder-only models like GPT, Claude, Llama, and Gemini dominate generative tasks because they unify understanding and generation in a single left-to-right pass. Encoder-decoder models like T5 and the original transformer remain effective for structured tasks like translation and summarization where distinct encoding and decoding phases are natural. The decoder-only design has won the scaling race because it simplifies training (next-token prediction), eliminates the need for explicit input-output separation, and scales more predictably.
| Model | Architecture | Parameters | Context length | Training data | Key innovation |
|---|---|---|---|---|---|
| Original Transformer (2017) | Encoder-decoder | 65M | 512 tokens | WMT translation | Self-attention replaces RNNs |
| BERT (2018) | Encoder-only | 110M–340M | 512 tokens | BooksCorpus + Wikipedia | Masked language modeling, bidirectional |
| GPT-3 (2020) | Decoder-only | 175B | 2,048 tokens | 300B tokens web corpus | Few-shot learning via scale |
| T5 (2020) | Encoder-decoder | 220M–11B | 512 tokens | C4 (750GB text) | Text-to-text unification |
| Llama 3 (2024) | Decoder-only | 8B–405B | 128K tokens | 15T+ tokens | Grouped-query attention, RoPE |
| Gemini (2024) | Decoder-only (multimodal) | Undisclosed | 1M+ tokens | Multimodal web-scale | Natively multimodal, long context |
| Mamba (2023) | State-space (non-transformer) | 130M–2.8B | Unlimited (linear) | Standard benchmarks | Selective state spaces, linear scaling |
```svg
```
**Scaling transformers reveals predictable power-law relationships between compute, data, and model quality.** The Chinchilla scaling laws (Hoffmann et al., 2022) showed that for a given compute budget, there is an optimal balance between model size and training tokens — training a smaller model on more data often outperforms training a larger model on less data. This insight shifted the field from simply making models bigger toward compute-optimal training. Modern frontier models train on 10 to 15 trillion tokens using thousands of GPUs or TPUs for months. The computational cost of training scales roughly as 6ND, where N is the number of parameters and D is the number of training tokens, counting both forward and backward passes. Inference cost, by contrast, depends primarily on the number of parameters and the sequence length, making model compression and efficient attention critical for deployment.
**Hardware design for transformers centers on dense matrix multiplication and memory bandwidth.** The attention mechanism and feed-forward layers are dominated by large matrix multiplications (GEMMs), making GPUs and TPUs with massive parallel multiply-accumulate arrays ideal. Training a single layer involves computing QKV projections, attention scores, output projections, and two FFN matrices — all GEMMs. The key bottleneck is often memory bandwidth rather than compute: moving weights from HBM to the compute units takes more time than the arithmetic itself, particularly during inference. This has driven architectural innovations like grouped-query attention (GQA), which reduces the KV cache size by sharing key-value heads across multiple query heads, and FlashAttention, which restructures the attention computation to minimize HBM reads by fusing operations in on-chip SRAM. Quantization from FP16 to INT8 or INT4 halves or quarters memory traffic while maintaining acceptable quality. These hardware-software co-design challenges explain why transformer inference has become the defining workload for AI chip design.
**Architectural variations continue to push the boundaries of what transformers can do.** Mixture-of-experts (MoE) models like Mixtral and Gemini activate only a subset of parameters for each token, achieving better quality per FLOP at the cost of higher total parameter count and memory. Speculative decoding uses a small draft model to propose multiple tokens that the larger model verifies in parallel, improving inference throughput. State-space models like Mamba challenge the transformer by replacing attention with linear-time recurrence, achieving competitive quality on some benchmarks with better scaling in sequence length. However, attention-based transformers continue to dominate at the frontier because their ability to route information dynamically between any two positions in a sequence — learned end-to-end — remains difficult to replicate with fixed-structure alternatives. The transformer is not just a model architecture; it is the computational substrate on which the current era of artificial intelligence is built.