multi-head latent attention

Multi-head latent attention (MLA) is the attention mechanism introduced in DeepSeek-V2 and carried into DeepSeek-V3, designed to shrink the KV cache without giving up the quality of full multi-head attention. Where grouped-query attention saves memory by having query heads share fewer key/value heads, MLA takes a different route: it compresses the keys and values of each token into a single small latent vector, caches only that latent, and reconstructs the per-head keys and values on the fly when attention runs. The result is a cache far smaller than even GQA, at the price of a little extra computation per step.\n\n**MLA caches a low-rank latent, not full keys and values.** For each token, MLA applies a down-projection that maps the hidden state into a compact latent vector, and it is this latent — not the full set of per-head K and V — that is written to the KV cache. When attention needs the keys and values, up-projection matrices expand the cached latent back into per-head K and V. Because the stored object is a single low-dimensional vector rather than two full-width tensors across every head, the per-token memory footprint drops by roughly an order of magnitude, which is exactly the quantity that limits context length and concurrency during generation.\n\n**A decoupled RoPE key keeps rotary position compatible with compression.** There is a catch: RoPE rotates keys by position, and you cannot cleanly apply a position-dependent rotation to a compressed latent and still fold the up-projection into the query and output weights. MLA solves this by splitting the key into two parts — a compressed content part reconstructed from the latent, and a small separate key that carries the RoPE rotation and is cached alongside the latent. This decoupling preserves relative-position behavior while keeping the bulk of the cache compressed, and it lets the up-projection matrices be absorbed into the query and output projections so the extra decompression adds only modest compute.\n\n| | MHA | GQA | MLA |\n|---|---|---|---|\n| What's cached | full K,V per head | K,V per group | one latent (+RoPE key) |\n| Cache size | 1× (largest) | ~4× smaller | ~10–15× smaller |\n| How it saves | — | share KV heads | low-rank compression |\n| Quality | baseline | ≈ MHA | ≈ MHA (often better) |\n| Extra cost | none | none | up-projection matmuls |\n| Origin | original Transformer | Llama/Mistral | DeepSeek-V2/V3 |\n\n```svg\n\n```\n\n**It trades a little compute for a lot of memory, which is the right trade for decoding.** Autoregressive generation is memory-bandwidth bound and capped by how many tokens' KV caches fit in GPU memory, so replacing full K/V storage with a small latent directly buys longer context and higher batch sizes. The up-projection does add matmuls, but decode has spare compute, and by absorbing the up-projection weights into the query and output matrices MLA keeps the overhead small. DeepSeek reported that MLA both cut the cache dramatically and matched or beat MHA quality, which is why it sits alongside GQA as one of the two dominant KV-reduction strategies and pairs naturally with MoE, PagedAttention, and long-context serving.\n\nRead MLA through a quant lens rather than a 'compress the cache' lens: the number it moves is bytes of KV cache per token, pushed down to the latent dimension plus a small RoPE key rather than two-times-heads-times-head-dim, roughly a 10-to-15-times reduction. The lever is the latent rank: smaller ranks cache less but discard more of the key/value detail, so the design question is how far you can compress before reconstruction loses the context the model needs — and MLA's result is that a surprisingly small latent, plus the decoupled positional key, recovers almost all of full attention's quality while spending only the decode-time compute you already had to spare.