attention mechanism

Attention is the core operation of the Transformer: it lets every token in a sequence look at every other token and pull in the information most relevant to it. Each token emits three learned vectors — a query, a key, and a value — and attention scores how well one token's query matches every token's key, turns those scores into weights that sum to one, and returns a weighted average of the values. The result is a context-aware representation of each token, computed in a single parallel operation rather than step by step.\n\n**A dot product measures relevance, a softmax turns it into a mixture.** For a given query, the model takes its dot product with every key in the sequence; a large dot product means that token is relevant. Those raw scores are divided by the square root of the head dimension to keep them from growing too large as vectors get wider, then passed through a softmax so they become non-negative weights that add to one. Attention is therefore a soft, differentiable lookup: instead of retrieving one entry from a table, it retrieves a blend of all entries, weighted by learned similarity.\n\n**The output is a weighted sum of values, and that is what makes context flow.** Once the weights are known, the output for each position is the sum of every token's value vector scaled by its weight. A token that needs its subject two words back will place most of its weight there; a token that needs a distant clause will reach across the whole sequence. Because queries, keys, and values are all learned projections, the model discovers on its own which relationships matter — subject-verb agreement, coreference, syntax — without any hard-wired notion of position or distance beyond what the encoding supplies.\n\n| Component | Role | Shape intuition |\n|---|---|---|\n| Query (Q) | what this token is looking for | one vector per token |\n| Key (K) | what each token offers | one vector per token |\n| Value (V) | the content to mix in | one vector per token |\n| QKᵀ | relevance score per pair | sequence × sequence |\n| ÷ √d + softmax | scores → weights (sum to 1) | row-wise distribution |\n| Σ weight·V | context-mixed output | one vector per token |\n\n```svg\n\n```\n\n**Its cost is quadratic in sequence length, and that shapes all of modern LLM hardware.** Because every query compares against every key, both the compute and the score matrix grow with the sequence length squared, and during generation the keys and values of all past tokens must be kept around — the KV cache. This single fact drives a huge amount of systems work: Flash Attention restructures the computation to avoid materializing the full score matrix, multi-query and grouped-query attention shrink the KV cache, and sparse or sliding-window variants skip most of the pairs outright. Self-attention (queries, keys, and values from the same sequence) builds internal context; cross-attention (queries from one sequence, keys and values from another) is how a decoder reads an encoder or a prompt.\n\nRead attention through a quant lens rather than a 'the model focuses' lens: the number that governs it is the sequence-length-squared count of query-key pairs, which sets both the FLOPs and the memory traffic every layer pays, and the KV cache turns that into a per-token memory cost during generation. Almost every attention variant is an attempt to move that number — reduce the pairs scored (sparse, sliding-window), reduce the bytes cached per token (MQA, GQA, MLA), or reduce the passes over memory (Flash Attention) — so the design question is always which part of the quadratic you can drop without losing the context the task actually needs.