attention

Attention is the Transformer operation that lets each token gather information from other tokens according to learned relevance rather than fixed distance. **Queries ask, keys match, and values carry content.** Scaled dot-product attention is $\text{softmax}\bigl(\frac{QK^\top}{\sqrt{d_k}}\bigr)V$: query-key products create scores, softmax turns them into weights, and those weights mix the value vectors. The $\sqrt{d_k}$ factor keeps scores numerically well behaved as the head dimension grows. **Masking determines what information is visible.** Encoders usually allow bidirectional attention, while autoregressive decoders use a causal mask so a token cannot see the future. Cross-attention uses queries from one sequence and keys and values from another. **The cost is memory as well as arithmetic.** Standard attention forms a sequence-by-sequence score matrix, giving quadratic growth with context length. FlashAttention computes the same result in tiles to reduce memory traffic, while grouped-query and multi-query attention shrink the key-value cache used during inference.