attention mechanism multi head

**Attention Mechanisms Beyond Vanilla (Multi-Head, Multi-Query, Grouped-Query, Sliding Window)** is **the evolution of transformer attention from the original scaled dot-product formulation to specialized variants that improve computational efficiency, memory usage, and long-context handling** — with each variant making different tradeoffs between representational capacity and inference speed. **Vanilla Scaled Dot-Product Attention** The foundational attention mechanism computes $ ext{Attention}(Q,K,V) = ext{softmax}(frac{QK^T}{sqrt{d_k}})V$ where queries (Q), keys (K), and values (V) are linear projections of input embeddings. Computational complexity is O(n²d) where n is sequence length and d is head dimension. Memory for storing the full attention matrix scales as O(n²), becoming the primary bottleneck for long sequences. The softmax operation creates a probability distribution over all positions, enabling global context aggregation. **Multi-Head Attention (MHA)** - **Parallel heads**: Input is projected into h parallel attention heads, each with dimension d_k = d_model/h (typically h=32, d_k=128 for large models) - **Diverse representations**: Each head can attend to different positions and learn different relationship types (syntactic, semantic, positional) - **Concatenation**: Head outputs are concatenated and projected through a linear layer to produce the final output - **KV cache**: During autoregressive inference, past key/value pairs for all heads are cached, consuming memory proportional to batch_size × n_heads × seq_len × d_k × 2 - **Standard usage**: Used in the original Transformer, BERT, GPT-2, and GPT-3 **Multi-Query Attention (MQA)** - **Shared KV projections**: All attention heads share a single set of key and value projections while maintaining separate query projections - **Memory reduction**: KV cache size reduced by factor of h (number of heads)—critical for high-throughput inference serving - **Speed improvement**: 3-10x faster inference with minimal quality degradation (typically <1% accuracy loss) - **Adoption**: Used in PaLM, Falcon, and StarCoder models - **Trade-off**: Slight reduction in model capacity due to shared representations, partially offset by faster training throughput enabling more tokens processed **Grouped-Query Attention (GQA)** - **Balanced approach**: Keys and values are shared within groups of heads rather than all heads or no heads - **Group count**: Typically 8 KV groups for 32 query heads (each KV group serves 4 query heads) - **Performance**: Achieves near-MHA quality with near-MQA efficiency—the best practical compromise - **Adoption**: LLaMA 2 (70B), Mistral, LLaMA 3, and most modern LLMs use GQA - **Uptraining from MHA**: Existing MHA models can be converted to GQA by mean-pooling adjacent KV heads and brief fine-tuning (5% of pretraining compute) **Sliding Window Attention (SWA)** - **Local attention**: Each token attends only to a fixed window of w surrounding tokens rather than the full sequence - **Linear complexity**: Computation scales as O(n × w) instead of O(n²), enabling processing of very long sequences - **Information propagation**: With L layers and window size w, information can propagate L × w positions through the network—sufficient for most tasks with adequate depth - **Mistral and Mixtral**: Use sliding window attention with w=4096 combined with full attention in selected layers - **Longformer pattern**: Combines sliding window (local) with global attention tokens (e.g., [CLS] token attends to all positions) for tasks requiring global context **Flash Attention and Hardware-Aware Implementations** - **IO-aware algorithm**: FlashAttention (Dao, 2022) computes exact attention without materializing the O(n²) attention matrix by tiling computation to fit in SRAM - **Speedup**: 2-4x faster than standard attention and uses O(n) memory instead of O(n²) - **FlashAttention-2**: Improved parallelism across sequence length and better work partitioning between CUDA warps, achieving 50-73% of theoretical peak FLOPS - **FlashAttention-3**: Leverages Hopper GPU features (TMA, FP8, warp specialization) for further speedup on H100s - **Universal adoption**: Now the default attention implementation in PyTorch, HuggingFace Transformers, and all major training frameworks **Emerging Attention Variants** - **Ring Attention**: Distributes attention computation across multiple devices by passing KV blocks in a ring topology, enabling near-infinite context lengths - **Linear attention**: Replaces softmax with kernel functions to achieve O(n) complexity but may sacrifice quality on tasks requiring precise attention patterns - **Differential attention**: Computes attention as the difference between two softmax attention maps, reducing noise and improving signal extraction - **Multi-head latent attention (MLA)**: DeepSeek-V2's approach that jointly compresses KV into a low-rank latent space, reducing KV cache by 93% while maintaining quality **The evolution of attention mechanisms reflects the fundamental tension between model expressiveness and computational practicality, with modern variants like GQA and Flash Attention enabling trillion-parameter models to serve billions of users at interactive speeds.**