attention mechanism transformer

**Attention Mechanisms** are the **neural network operations that dynamically compute weighted combinations of value vectors based on query-key similarity — enabling each element in a sequence to gather information from all other elements based on relevance, forming the computational core of transformer architectures and the single most impactful innovation in modern deep learning**. **Scaled Dot-Product Attention** The fundamental operation: Attention(Q, K, V) = softmax(QKᵀ/√dₖ)V where Q (queries), K (keys), V (values) are linear projections of the input. The dot product QKᵀ computes pairwise similarity between all query-key pairs, softmax normalizes to a probability distribution, and the result weights the values. The √dₖ scaling prevents attention scores from becoming extreme in high dimensions. **Multi-Head Attention** Instead of one attention function with d-dimensional keys, queries, and values, the computation splits into h parallel heads, each with dₖ=d/h dimensions. Each head can attend to different aspects of the input (syntactic structure, semantic similarity, positional relationships). The concatenated head outputs are linearly projected to produce the final output. **Self-Attention vs. Cross-Attention** - **Self-Attention**: Q, K, V all derive from the same sequence. Each token attends to every other token in the same sequence. Used in encoder layers and decoder masked self-attention. - **Cross-Attention**: Q comes from one sequence (decoder), K and V from another (encoder output). Enables the decoder to attend to relevant encoder positions. Used in encoder-decoder models, VLMs (text queries attend to visual features), and diffusion U-Nets (visual features attend to text conditioning). - **Causal (Masked) Attention**: A mask prevents tokens from attending to future positions: attention_mask[i][j] = -∞ for j > i. Essential for autoregressive generation. **KV Cache** During autoregressive inference, each new token only needs its own query vector — the keys and values from all previous tokens are cached and reused. This reduces per-token computation from O(N²) to O(N) but requires O(N × L × d) memory that grows with sequence length. KV cache memory management is the primary bottleneck for long-context LLM serving. **Efficient Attention Variants** - **Flash Attention**: Fuses the attention computation into a single GPU kernel that operates on tiles of Q, K, V in SRAM, avoiding materialization of the N×N attention matrix in HBM. Reduces memory from O(N²) to O(N) and achieves 2-4x wall-clock speedup. The default attention implementation in all modern frameworks. - **Multi-Query Attention (MQA)**: All heads share a single K and V projection — reduces KV cache size by h× with minor quality loss. - **Grouped-Query Attention (GQA)**: Groups of heads share K/V projections (e.g., 8 groups for 32 heads = 4x KV cache reduction). Used in LLaMA 2 70B, Mistral, and most production LLMs as the sweet spot between MHA and MQA. Attention Mechanisms are **the core computation that makes transformers transformers** — the dynamic, content-dependent information routing that replaced fixed convolution kernels and recurrent state updates with a universally flexible mechanism for relating any part of the input to any other.