transformer architecture

**Transformer** — the neural network architecture based entirely on attention mechanisms that replaced RNNs and became the foundation of modern AI (GPT, BERT, ViT, Stable Diffusion). **Architecture** - **Encoder**: Processes input sequence → produces contextual representations. Used in BERT, ViT - **Decoder**: Generates output token-by-token using masked self-attention. Used in GPT - **Encoder-Decoder**: Both components. Used in T5, BART, original machine translation **Key Components (per layer)** 1. **Multi-Head Self-Attention**: Each token attends to all others 2. **Feed-Forward Network (FFN)**: Two linear layers with activation (processes each position independently) 3. **Layer Normalization**: Stabilizes training 4. **Residual Connections**: $output = LayerNorm(x + SubLayer(x))$ **Positional Encoding** - Transformers have no built-in notion of order (unlike RNNs) - Must add position information: sinusoidal (original), learned, RoPE (rotary — used in LLaMA/GPT-NeoX) **Scale** - GPT-3: 96 layers, 175B parameters - GPT-4: Estimated 1.8T parameters (MoE) - Each layer: ~$12d^2$ parameters (for hidden dimension $d$) **The Transformer** is arguably the most important architecture in AI history — it unified NLP, vision, audio, and multimodal AI under one framework.