multi-query attention

**Multi-Query and Grouped Query Attention (GQA)** are **attention variants that share key-value representations across multiple query heads — reducing KV cache memory by 8-16x and decoder-only inference latency by 25-40% while maintaining near-identical quality to standard multi-head attention**. **Standard Multi-Head Attention Baseline:** - **Head Structure**: Q, K, V each split into h heads (h=32 for 1B models, h=96 for 70B) with dimension d_k = d_model/h - **Attention Computation**: each head independently computes Attention(Q_i, K_i, V_i) = softmax(Q_i·K_i^T/√d_k)·V_i - **Parameter Count**: queries, keys, values each contain h×d_k = d_model parameters — full matrix multiplications - **KV Cache Size**: storing K, V for all previous tokens creates matrix [seq_len, h, d_k] — 70B Llama with 32K context requires 78GB per batch **Multi-Query Attention (MQA) Architecture:** - **Single KV Head**: using single K, V across all Q heads: Attention(Q_i, K, V) where K, V ∈ ℝ^(seq_len × d_k) - **Parameter Reduction**: reducing K, V parameters from h×d_k to d_k — 96x reduction for 96-head models - **KV Cache Reduction**: memory from [seq_len, h, d_k] to [seq_len, d_k] — 96x reduction (78GB→0.8GB for 70B model) - **Quality Trade-off**: 1-2% accuracy loss on benchmarks compared to standard attention — minimal impact on downstream performance - **Inference Speedup**: memory bandwidth bottleneck becomes compute-bound, latency 25-35% faster — especially dramatic for long sequences **Grouped Query Attention (GQA) - Balanced Approach:** - **Intermediate Grouping**: using g query heads per key-value head (g=4-8 typical) instead of h heads - **Flexibility**: scaling from MQA (g=1) to standard attention (g=h) with continuous parameter-quality trade-off - **Common Configurations**: h=64 query heads, g=8 key-value heads (8x KV reduction) — standard in Llama 2, Mistral models - **Quality Performance**: with g=8, achieving 99.5% quality of standard attention while reducing KV cache 8x — empirically better than MQA - **Adoption**: Llama 2 70B uses GQA by default with 8-head groups — production standard for modern models **Mathematical Formulation:** - **GQA Attention**: Attention(Q_{i,j}, K_i, V_i) where i ∈ [0, g), j ∈ [0, h/g) groups queries by key-value head - **Broadcasting**: each of g key-value heads broadcasts to h/g query heads — implemented as reshape and expand operations - **Gradient Flow**: gradients from all query heads in group accumulate to single key-value head — implicit head collaboration - **Attention Pattern**: each key-value head attends to same token positions across all grouped query heads — enables more expressive attention **Inference Optimization Impact:** - **Memory Bandwidth**: decoder latency bottleneck shifts from KV cache access (100GB/s bandwidth) to compute (312 TFLOPS peak) - **Batch Size Scaling**: with MQA/GQA, batch size increases 8-16x before KV cache OOM — servers handle 10x more concurrent requests - **Prefill-Decode Overlap**: GQA enables more efficient pipeline overlap (prefill on compute cores, decode from cache) — 30-50% throughput improvement - **Long Context**: GQA enables 100K+ context windows on single GPU (Llama 2 Long on 80GB A100) — infeasible with standard attention **Practical Deployment Benefits:** - **Latency Reduction**: 70B Llama 2 goes from 120ms to 80-90ms first-token latency with GQA — critical for interactive applications - **Throughput**: serving platform throughput increases from 50 req/s to 150-200 req/s per GPU — 3-4x improvement - **Cost**: fewer GPUs needed for same throughput (200→50 GPUs for 1000 req/s) — 75% cost reduction - **Mobile Deployment**: GQA enables running 13B models on edge devices with KV cache fitting in 8GB DRAM **Model Architecture Adoption:** - **Llama 2 Family**: all models (7B, 13B, 70B) use GQA with g=8 groups — standardized across Meta models - **Mistral 7B**: uses GQA for efficiency, enabling strong performance with fewer parameters than Llama - **Falcon 40B**: adopts GQA achieving Llama 70B quality with 40% fewer parameters - **GPT-style Models**: OpenAI models still use standard attention (possibly using MQA internally) — GQA benefits still untapped for API models **Advanced Techniques:** - **Grouped Query with Recomputation**: storing only g key-value heads, recomputing intermediate query-head values during backward pass — reduces cache memory further - **Dynamic Head Grouping**: adaptively grouping based on attention pattern sparsity per layer — compute-aware optimization - **Cross-Attention Variants**: applying GQA to encoder-decoder cross-attention for 4-8x reduction — enables larger batch sizes in sequence-to-sequence models - **Hybrid Approaches**: using GQA in early layers (lower precision) and standard attention in final layers — balances quality and efficiency **Multi-Query and Grouped Query Attention are transforming LLM inference economics — enabling practical deployment of large models through 8-16x KV cache reduction while maintaining 99%+ quality compared to standard multi-head attention.**