attention mechanism transformer

**The Attention Mechanism** is the **core computational primitive of the Transformer architecture that enables each token in a sequence to dynamically gather information from all other tokens based on learned relevance scores — computing a weighted combination of value vectors where the weights are determined by the compatibility between query and key vectors, forming the foundation of virtually all modern language models, vision models, and multimodal AI systems**. **Scaled Dot-Product Attention** Given input embeddings X, three linear projections produce: - **Queries (Q)**: What information each token is looking for. - **Keys (K)**: What information each token offers. - **Values (V)**: The actual information content. Attention(Q, K, V) = softmax(Q * K^T / sqrt(d_k)) * V The dot product Q*K^T computes pairwise compatibility scores. Division by sqrt(d_k) prevents the softmax from saturating into one-hot vectors for large dimension d_k. The softmax normalizes scores into a probability distribution. Multiplying by V produces a weighted sum of value vectors. **Multi-Head Attention** Instead of computing a single attention function, the model runs H parallel attention heads (typically 8-128), each with its own Q/K/V projections of dimension d_k = d_model/H. Each head can attend to different aspects of the input (syntactic relationships, semantic similarity, positional patterns). The head outputs are concatenated and linearly projected. **Causal (Autoregressive) Attention** For language generation, a causal mask prevents each token from attending to future positions — token i can only see tokens 1 through i. This is implemented by setting the upper-triangular entries of the attention matrix to -infinity before softmax. **KV Cache** During autoregressive generation, previously computed key and value vectors don't change as new tokens are generated. The KV cache stores all past K and V vectors, so each new token only computes its own Q and attends to the cached K/V. This reduces per-token computation from O(n²) to O(n) but requires memory that grows linearly with sequence length. **Efficiency Optimizations** - **Flash Attention**: Fuses the attention computation into a single GPU kernel that never materializes the full n×n attention matrix in HBM. Achieves 2-4x speedup and enables much longer sequences by reducing memory from O(n²) to O(n). - **Multi-Query Attention (MQA)**: All heads share the same K and V projections (only Q differs per head). Reduces KV cache size by H×, dramatically improving inference throughput. - **Grouped-Query Attention (GQA)**: A compromise where K/V are shared among groups of heads (e.g., 8 KV heads for 32 query heads). Used in LLaMA 2, Mistral, and most modern LLMs. - **Sliding Window Attention**: Each token attends only to the nearest W tokens (e.g., W=4096), giving O(n*W) complexity. Combined with a few global attention layers, this handles very long sequences. The Attention Mechanism is **the algorithm that taught neural networks to focus** — replacing fixed-pattern information routing with dynamic, content-dependent communication that adapts to every input, enabling the unprecedented generality of modern AI.