Grouped Query Attention (GQA) is the attention mechanism that shares key and value projections across groups of query heads, interpolating between multi-head attention (MHA) and multi-query attention (MQA) — reducing KV cache size by 4-8× while maintaining 95-99% of MHA quality, used in Llama 2, Mistral, and other modern LLMs to enable efficient long-context inference within memory constraints.

GQA Architecture:

• Head Grouping: divides H query heads into G groups; each group shares single K and V head; group size H/G typically 4-8; example: Llama 2 70B uses 64 query heads with 8 KV heads (8 groups of 8 queries each)
• Projection Dimensions: query projection Q has dimension d_model → H×d_head; key and value projections K, V have dimension d_model → G×d_head where G<
• Attention Computation: within each group, all query heads attend using same K and V; mathematically: Attention_i(Q_i, K_g, V_g) where i is query head index, g=floor(i×G/H) is group index; output heads concatenated as in standard MHA
• KV Cache: during autoregressive generation, caches G×L×d_head values instead of H×L×d_head for MHA; at L=4096 tokens, H=64 heads, d_head=128, G=8: cache size 4MB vs 32MB for MHA (8× reduction)

Comparison with MHA and MQA:

• Multi-Head Attention (MHA): each query head has dedicated K and V heads; maximum expressiveness but largest KV cache; standard in BERT, GPT-2, GPT-3; KV cache size H×L×d_head
• Multi-Query Attention (MQA): all query heads share single K and V head (G=1); smallest KV cache (H× reduction) but 1-3% quality degradation vs MHA; used in PaLM, Falcon; can cause attention head collapse where heads learn similar patterns
• GQA Sweet Spot: G=4-8 provides 4-8× cache reduction with <0.5% quality loss vs MHA; avoids head collapse of MQA; Llama 2 experiments show G=8 matches MHA perplexity while enabling 8× longer contexts in same memory
• Training Cost: GQA reduces KV projection FLOPs by H/G but attention computation unchanged; total training speedup 5-10% for typical models; inference speedup 10-20% from reduced memory bandwidth

Implementation and Training:

• Uptraining from MQA: can convert MQA checkpoint to GQA by replicating single KV head G times; continue training for 5-10% of original tokens; achieves MHA-level quality; used in Llama 2 development (MQA pretrain → GQA uptrain)
• Direct GQA Training: train with GQA from scratch; no quality loss vs MHA; slightly faster training than MHA; requires framework support (PyTorch 2.0+, Hugging Face Transformers, Megatron-LM)
• Group Size Selection: G=H/8 works well for most models; larger models benefit from more groups (G=8-16); smaller models can use fewer (G=2-4); diminishing returns beyond G=16
• Attention Mask Handling: same mask applied to all heads in group; no special handling needed; compatible with causal masking, padding masks, and attention biases

Memory and Performance Impact:

• KV Cache Scaling: for model with H heads, sequence length L, head dimension d, batch size B: MHA cache = B×H×L×d×2 bytes; GQA cache = B×G×L×d×2 bytes; at B=32, H=32, G=4, L=4096, d=128: GQA uses 128MB vs 512MB for MHA
• Batch Size Increase: reduced KV cache enables 4-8× larger batch sizes in same memory; improves throughput by 3-6× for inference serving; critical for high-throughput deployment
• Context Length: at fixed memory budget, GQA enables H/G× longer contexts; Llama 2 70B with GQA (G=8) handles 32K tokens in 80GB vs 4K for MHA; enables long-document applications
• Bandwidth Reduction: loading KV cache from HBM is major bottleneck in autoregressive decoding; GQA reduces bandwidth by H/G; improves tokens/second by 15-25% on memory-bound workloads

Adoption in Production Models:

• Llama 2: 7B, 13B, 70B all use GQA with G=8 (except 7B uses MQA); enables 4K default context, 32K with RoPE scaling; key factor in Llama 2's efficiency advantage over Llama 1
• Mistral 7B: uses GQA with G=8; achieves GPT-3.5-level quality at 7B parameters; 8K context window; sliding window attention combined with GQA enables efficient long-context processing
• Code Llama: inherits GQA from Llama 2; 16K-100K context variants possible due to GQA memory efficiency; critical for code completion and repository-level understanding
• Falcon 180B: uses MQA (G=1) for maximum cache reduction; accepts small quality trade-off for deployment efficiency; demonstrates viability of extreme cache reduction at large scale

Grouped Query Attention is the practical compromise that makes long-context LLMs deployable — by carefully balancing model quality and memory efficiency, GQA has become the standard attention mechanism for modern LLMs, enabling the multi-turn conversations and long-document processing that define production AI applications.