Attention Mechanisms Beyond Vanilla (Multi-Head, Multi-Query, Grouped-Query, Sliding Window) is the evolution of transformer attention from the original scaled dot-product formulation to specialized variants that improve computational efficiency, memory usage, and long-context handling — with each variant making different tradeoffs between representational capacity and inference speed.

Vanilla Scaled Dot-Product Attention

The foundational attention mechanism computes $ ext{Attention}(Q,K,V) = ext{softmax}(frac{QK^T}{sqrt{d_k}})V$ where queries (Q), keys (K), and values (V) are linear projections of input embeddings. Computational complexity is O(n²d) where n is sequence length and d is head dimension. Memory for storing the full attention matrix scales as O(n²), becoming the primary bottleneck for long sequences. The softmax operation creates a probability distribution over all positions, enabling global context aggregation.

Multi-Head Attention (MHA)

• Parallel heads: Input is projected into h parallel attention heads, each with dimension d_k = d_model/h (typically h=32, d_k=128 for large models)
• Diverse representations: Each head can attend to different positions and learn different relationship types (syntactic, semantic, positional)
• Concatenation: Head outputs are concatenated and projected through a linear layer to produce the final output
• KV cache: During autoregressive inference, past key/value pairs for all heads are cached, consuming memory proportional to batch_size × n_heads × seq_len × d_k × 2
• Standard usage: Used in the original Transformer, BERT, GPT-2, and GPT-3

Multi-Query Attention (MQA)

• Shared KV projections: All attention heads share a single set of key and value projections while maintaining separate query projections
• Memory reduction: KV cache size reduced by factor of h (number of heads)—critical for high-throughput inference serving
• Speed improvement: 3-10x faster inference with minimal quality degradation (typically <1% accuracy loss)
• Adoption: Used in PaLM, Falcon, and StarCoder models
• Trade-off: Slight reduction in model capacity due to shared representations, partially offset by faster training throughput enabling more tokens processed

Grouped-Query Attention (GQA)

• Balanced approach: Keys and values are shared within groups of heads rather than all heads or no heads
• Group count: Typically 8 KV groups for 32 query heads (each KV group serves 4 query heads)
• Performance: Achieves near-MHA quality with near-MQA efficiency—the best practical compromise
• Adoption: LLaMA 2 (70B), Mistral, LLaMA 3, and most modern LLMs use GQA
• Uptraining from MHA: Existing MHA models can be converted to GQA by mean-pooling adjacent KV heads and brief fine-tuning (5% of pretraining compute)

Sliding Window Attention (SWA)

• Local attention: Each token attends only to a fixed window of w surrounding tokens rather than the full sequence
• Linear complexity: Computation scales as O(n × w) instead of O(n²), enabling processing of very long sequences
• Information propagation: With L layers and window size w, information can propagate L × w positions through the network—sufficient for most tasks with adequate depth
• Mistral and Mixtral: Use sliding window attention with w=4096 combined with full attention in selected layers
• Longformer pattern: Combines sliding window (local) with global attention tokens (e.g., [CLS] token attends to all positions) for tasks requiring global context

Flash Attention and Hardware-Aware Implementations

• IO-aware algorithm: FlashAttention (Dao, 2022) computes exact attention without materializing the O(n²) attention matrix by tiling computation to fit in SRAM
• Speedup: 2-4x faster than standard attention and uses O(n) memory instead of O(n²)
• FlashAttention-2: Improved parallelism across sequence length and better work partitioning between CUDA warps, achieving 50-73% of theoretical peak FLOPS
• FlashAttention-3: Leverages Hopper GPU features (TMA, FP8, warp specialization) for further speedup on H100s
• Universal adoption: Now the default attention implementation in PyTorch, HuggingFace Transformers, and all major training frameworks

Emerging Attention Variants

• Ring Attention: Distributes attention computation across multiple devices by passing KV blocks in a ring topology, enabling near-infinite context lengths
• Linear attention: Replaces softmax with kernel functions to achieve O(n) complexity but may sacrifice quality on tasks requiring precise attention patterns
• Differential attention: Computes attention as the difference between two softmax attention maps, reducing noise and improving signal extraction
• Multi-head latent attention (MLA): DeepSeek-V2's approach that jointly compresses KV into a low-rank latent space, reducing KV cache by 93% while maintaining quality

The evolution of attention mechanisms reflects the fundamental tension between model expressiveness and computational practicality, with modern variants like GQA and Flash Attention enabling trillion-parameter models to serve billions of users at interactive speeds.