Attention Mechanisms in Transformers are the core computational primitive that enables each token in a sequence to dynamically weight and aggregate information from all other tokens based on learned relevance — replacing fixed convolution windows and recurrent state with flexible, content-dependent information routing that captures arbitrary-range dependencies in a single layer.

Scaled Dot-Product Attention:

• Query-Key-Value Framework: input X is projected into three matrices: Q (queries), K (keys), V (values) through learned linear projections; attention computes Attention(Q,K,V) = softmax(QK^T/√d_k)·V where d_k is the key dimension
• Scaling Factor: division by √d_k prevents dot products from growing too large with increasing dimension, which would push softmax into extreme saturation regions with vanishing gradients; without scaling, training becomes unstable for d_k > 64
• Attention Matrix: QK^T produces an N×N attention matrix (N = sequence length) where each entry represents the relevance between a query token and all key tokens; softmax normalizes each row to form a probability distribution over keys
• Causal Masking: for autoregressive (decoder) models, mask upper triangle of attention matrix with -∞ before softmax; ensures token i can only attend to tokens j ≤ i, preventing information leakage from future tokens during training and generation

Multi-Head Attention:

• Parallel Heads: instead of single attention with d_model dimensions, split into h parallel heads (h=8-32) with d_k = d_model/h each; each head learns different attention patterns (positional, syntactic, semantic relationships)
• Head Specialization: empirically, different heads attend to different aspects — some capture nearby tokens (local syntax), others capture distant dependencies (long-range coreference), some specialize on specific token types (punctuation, entities)
• Output Projection: concatenate all head outputs and project through W_O (d_model × d_model); this output projection mixes information across heads, enabling complex interaction patterns that no single head could capture
• Grouped Query Attention (GQA): groups of query heads share the same key and value heads; reduces KV cache memory by 4-8× (Llama 2 70B uses 8 KV heads shared across 64 query heads); minimal quality reduction vs full multi-head attention

Cross-Attention:

• Encoder-Decoder Coupling: queries come from the decoder, keys and values come from the encoder output; enables the decoder to attend to relevant encoder positions when generating each output token
• Text-to-Image: in diffusion models (Stable Diffusion), cross-attention injects text conditioning; queries from the U-Net spatial features, keys/values from CLIP text embeddings; controls which image regions correspond to which text tokens
• Multi-Modal Fusion: cross-attention between vision and language streams enables visual question answering, image captioning, and multimodal reasoning; the attention matrix reveals which visual regions the model considers when generating each word

Optimization and Efficiency:

• Flash Attention: fused kernel that computes attention in tiles, never materializing the full N×N attention matrix in HBM; reduces memory from O(N²) to O(N) and achieves 2-4× speedup by minimizing HBM reads/writes; the standard implementation in all modern training frameworks
• KV Cache: during autoregressive generation, cache previously computed key and value vectors; each new token only computes its own Q and attends to cached K,V; reduces per-token computation from O(N²) to O(N) but requires O(N·d·layers) memory
• Paged Attention (vLLM): manages KV cache using virtual memory paging — allocates KV cache in non-contiguous blocks, eliminating memory fragmentation and enabling efficient batch serving with variable-length sequences
• Multi-Query Attention (MQA): all query heads share a single key and single value head; most extreme KV cache compression (1/h of standard MHA); used in PaLM and Falcon; trades some quality for massive inference efficiency

Attention mechanisms are the computational heart of the Transformer revolution — their ability to dynamically route information based on content rather than position has made them the universal building block of modern AI, powering language models, vision transformers, protein structure prediction, and every major AI breakthrough since 2017.