grouped query attention explained
**Multi-query and grouped-query attention are two ways of shrinking the KV cache itself, by making multiple attention heads share the same keys and values instead of each head keeping its own.** The attention mechanism covered previously runs many attention "heads" in parallel inside each layer, each one learning to focus on different kinds of relationships between tokens. In the original transformer design, every one of those heads computes and caches its own separate key and value vectors — which means the KV cache's size multiplies by the number of heads, on top of already growing with every token generated. Multi-query and grouped-query attention attack that multiplication directly, because it turns out the queries need to stay diverse across heads, but the keys and values they're compared against don't have to be.
**Multi-query attention (MQA) is the aggressive version: every head keeps its own query, but all heads share one single set of keys and values.** Instead of storing a separate key/value pair per head, the model stores just one — and every head's distinct query is compared against that same shared pair. This can shrink KV cache size by roughly the number of heads (commonly 8, 16, or more), which is a dramatic memory-bandwidth win for decode. The tradeoff is that sharing one key/value set across every head removes some of the representational diversity multi-head attention was designed to provide, which can show up as a small quality cost, particularly on tasks needing very fine-grained distinctions between heads.
```svg
```
**Grouped-query attention (GQA) sits deliberately between those two extremes, and it's why most current large models use it rather than full MQA.** GQA splits the attention heads into a handful of groups — say, 8 groups out of 32 total heads — and every head within a group shares one key/value pair, while different groups keep their own. This recovers most of MQA's memory savings (the KV cache still shrinks by roughly the group size, not just staying at full size) while keeping meaningfully more representational diversity than collapsing everything down to a single shared key/value pair. The group count becomes a tunable dial: more groups means closer to full multi-head quality and a bigger cache; fewer groups means closer to MQA's memory savings and a slightly higher quality cost.
| Attention Variant | Key/Value Sharing | Relative KV Cache Size | Quality vs. Standard Multi-Head |
|---|---|---|---|
| Multi-Head (MHA) | None — every head has its own | Largest (1x per head) | Baseline |
| Grouped-Query (GQA) | Shared within small groups of heads | Reduced by group size | Very close to baseline |
| Multi-Query (MQA) | Shared across all heads | Smallest (1x total) | Small measurable drop |
```flowchart
st=>start: Model architecture decides how many heads share each key/value pair
mha=>operation: Standard multi-head — every head keeps its own key/value, cache scales with head count
gqa=>operation: Grouped-query — heads split into groups, each group shares one key/value pair
mqa=>operation: Multi-query — every head shares a single key/value pair across the whole layer
cache=>operation: Chosen scheme fixes how many key/value sets the KV cache must store per token
decode=>operation: Smaller per-token cache means less memory traffic per decode step
pass=>end: Faster or higher-throughput serving, traded against head-diversity quality cost
st->mha->gqa->mqa->cache->decode->pass
```
**This is the architectural counterpart to the quantization discussion, and the two combine directly in real deployed systems.** Quantization shrinks how many bytes each stored key, value, and weight takes; grouped-query and multi-query attention shrink how many separate key/value sets there are to store in the first place. Applying both together compounds the savings, which is exactly why a serving-optimized AI accelerator's practical throughput depends as much on which attention variant a model was trained with as on the accelerator's raw peak FLOPs or memory bandwidth spec — the model architecture and the hardware are solving the same memory-bandwidth problem from two different directions.