Multi-Query and Grouped Query Attention (GQA) are attention variants that share key-value representations across multiple query heads — reducing KV cache memory by 8-16x and decoder-only inference latency by 25-40% while maintaining near-identical quality to standard multi-head attention.

Standard Multi-Head Attention Baseline:

• Head Structure: Q, K, V each split into h heads (h=32 for 1B models, h=96 for 70B) with dimension d_k = d_model/h
• Attention Computation: each head independently computes Attention(Q_i, K_i, V_i) = softmax(Q_i·K_i^T/√d_k)·V_i
• Parameter Count: queries, keys, values each contain h×d_k = d_model parameters — full matrix multiplications
• KV Cache Size: storing K, V for all previous tokens creates matrix [seq_len, h, d_k] — 70B Llama with 32K context requires 78GB per batch

Multi-Query Attention (MQA) Architecture:

• Single KV Head: using single K, V across all Q heads: Attention(Q_i, K, V) where K, V ∈ ℝ^(seq_len × d_k)
• Parameter Reduction: reducing K, V parameters from h×d_k to d_k — 96x reduction for 96-head models
• KV Cache Reduction: memory from [seq_len, h, d_k] to [seq_len, d_k] — 96x reduction (78GB→0.8GB for 70B model)
• Quality Trade-off: 1-2% accuracy loss on benchmarks compared to standard attention — minimal impact on downstream performance
• Inference Speedup: memory bandwidth bottleneck becomes compute-bound, latency 25-35% faster — especially dramatic for long sequences

Grouped Query Attention (GQA) - Balanced Approach:

• Intermediate Grouping: using g query heads per key-value head (g=4-8 typical) instead of h heads
• Flexibility: scaling from MQA (g=1) to standard attention (g=h) with continuous parameter-quality trade-off
• Common Configurations: h=64 query heads, g=8 key-value heads (8x KV reduction) — standard in Llama 2, Mistral models
• Quality Performance: with g=8, achieving 99.5% quality of standard attention while reducing KV cache 8x — empirically better than MQA
• Adoption: Llama 2 70B uses GQA by default with 8-head groups — production standard for modern models

Mathematical Formulation:

• GQA Attention: Attention(Q_{i,j}, K_i, V_i) where i ∈ [0, g), j ∈ [0, h/g) groups queries by key-value head
• Broadcasting: each of g key-value heads broadcasts to h/g query heads — implemented as reshape and expand operations
• Gradient Flow: gradients from all query heads in group accumulate to single key-value head — implicit head collaboration
• Attention Pattern: each key-value head attends to same token positions across all grouped query heads — enables more expressive attention

Inference Optimization Impact:

• Memory Bandwidth: decoder latency bottleneck shifts from KV cache access (100GB/s bandwidth) to compute (312 TFLOPS peak)
• Batch Size Scaling: with MQA/GQA, batch size increases 8-16x before KV cache OOM — servers handle 10x more concurrent requests
• Prefill-Decode Overlap: GQA enables more efficient pipeline overlap (prefill on compute cores, decode from cache) — 30-50% throughput improvement
• Long Context: GQA enables 100K+ context windows on single GPU (Llama 2 Long on 80GB A100) — infeasible with standard attention

Practical Deployment Benefits:

• Latency Reduction: 70B Llama 2 goes from 120ms to 80-90ms first-token latency with GQA — critical for interactive applications
• Throughput: serving platform throughput increases from 50 req/s to 150-200 req/s per GPU — 3-4x improvement
• Cost: fewer GPUs needed for same throughput (200→50 GPUs for 1000 req/s) — 75% cost reduction
• Mobile Deployment: GQA enables running 13B models on edge devices with KV cache fitting in 8GB DRAM

Model Architecture Adoption:

• Llama 2 Family: all models (7B, 13B, 70B) use GQA with g=8 groups — standardized across Meta models
• Mistral 7B: uses GQA for efficiency, enabling strong performance with fewer parameters than Llama
• Falcon 40B: adopts GQA achieving Llama 70B quality with 40% fewer parameters
• GPT-style Models: OpenAI models still use standard attention (possibly using MQA internally) — GQA benefits still untapped for API models

Advanced Techniques:

• Grouped Query with Recomputation: storing only g key-value heads, recomputing intermediate query-head values during backward pass — reduces cache memory further
• Dynamic Head Grouping: adaptively grouping based on attention pattern sparsity per layer — compute-aware optimization
• Cross-Attention Variants: applying GQA to encoder-decoder cross-attention for 4-8x reduction — enables larger batch sizes in sequence-to-sequence models
• Hybrid Approaches: using GQA in early layers (lower precision) and standard attention in final layers — balances quality and efficiency

Multi-Query and Grouped Query Attention are transforming LLM inference economics — enabling practical deployment of large models through 8-16x KV cache reduction while maintaining 99%+ quality compared to standard multi-head attention.