Grouped-Query Attention (GQA) is an attention architecture that provides a tunable middle ground between Multi-Head Attention (MHA) and Multi-Query Attention (MQA) — using G groups of KV heads (where each group serves multiple query heads) to achieve near-MQA inference speed with near-MHA quality, making it the recommended default for new LLM architectures as adopted by Llama-2 70B, Mistral, Gemma, and most modern open-source models.

What Is GQA?

• Definition: GQA (Ainslie et al., 2023) partitions the H query heads into G groups, with each group sharing a single set of Key and Value projections. When G=1, it's MQA. When G=H, it's standard MHA. Values in between provide a configurable quality-speed trade-off.
• The Motivation: MQA (1 KV head) is very fast but shows quality degradation on complex reasoning tasks. MHA (H KV heads) preserves quality but has an enormous KV-cache. GQA finds the sweet spot — typically 8 KV groups for 64 query heads gives ~95% of MHA quality at ~90% of MQA speed.
• Practical Default: GQA has become the de facto standard for new LLM architectures because it provides the best quality-speed Pareto curve.

Architecture Visualization

MHA:  Q₁ Q₂ Q₃ Q₄ Q₅ Q₆ Q₇ Q₈   (8 query heads)
      K₁ K₂ K₃ K₄ K₅ K₆ K₇ K₈   (8 KV heads — one per query)

GQA:  Q₁ Q₂ Q₃ Q₄ Q₅ Q₆ Q₇ Q₈   (8 query heads)
      K₁ K₁ K₂ K₂ K₃ K₃ K₄ K₄   (4 KV groups — shared pairs)

MQA:  Q₁ Q₂ Q₃ Q₄ Q₅ Q₆ Q₇ Q₈   (8 query heads)
      K₁ K₁ K₁ K₁ K₁ K₁ K₁ K₁   (1 KV head — shared by all)

KV-Cache Comparison

Converting MHA Checkpoints to GQA

One key advantage: existing MHA models can be converted to GQA by mean-pooling the KV heads within each group and continuing training (uptraining). This avoids training from scratch.

# Convert 64 KV heads → 8 groups
# Each group = mean of 8 consecutive KV heads
group_1_K = mean(K_1, K_2, ..., K_8)
group_2_K = mean(K_9, K_10, ..., K_16)
...
# Then uptrain for ~5% of original training tokens

Models Using GQA

Grouped-Query Attention is the recommended default attention architecture for modern LLMs — providing a configurable KV-cache reduction (4-8× typical) that preserves near-full MHA quality while approaching MQA inference speeds, with the additional advantage of being convertible from existing MHA checkpoints through mean-pooling and uptraining rather than requiring training from scratch.