attention mechanism transformer

**Attention Mechanisms** are the **neural network components that dynamically weight the importance of different input elements relative to a query — enabling models to selectively focus on relevant information regardless of positional distance, forming the computational foundation of the Transformer architecture that powers all modern language models, vision transformers, and multimodal AI systems**. **The Core Computation** Scaled dot-product attention: Attention(Q, K, V) = softmax(QK^T / √d_k) × V Where Q (queries), K (keys), and V (values) are linear projections of the input. QK^T computes similarity scores between all query-key pairs. Softmax normalizes scores to attention weights. The output is a weighted sum of values. **Multi-Head Attention (MHA)** Instead of one attention function, project Q, K, V into h separate subspaces (heads), compute attention independently in each, then concatenate and project: MultiHead(Q, K, V) = Concat(head_1, ..., head_h) × W_O where head_i = Attention(Q×W_Qi, K×W_Ki, V×W_Vi) Each head can attend to different aspects — one head might capture syntactic relationships (subject-verb), another semantic similarity, another positional patterns. Standard: h=8-128 heads, d_k = d_model/h. **Attention Variants** - **Self-Attention**: Q, K, V all derived from the same input sequence. Each token attends to all tokens in the same sequence. Used in both encoder (bidirectional) and decoder (causal/masked). - **Cross-Attention**: Q from one sequence (decoder), K/V from another (encoder). The mechanism that connects encoder representations to decoder generation in encoder-decoder models (translation, image captioning, speech recognition). - **Causal (Masked) Attention**: In autoregressive generation, token i can only attend to tokens 1..i (not future tokens). Implemented by setting upper-triangular attention scores to -∞ before softmax. **Efficient Attention Variants** Standard attention is O(n²) in sequence length — prohibitive for long sequences: - **Flash Attention**: Reorders the attention computation to minimize HBM (GPU memory) reads/writes by computing attention in tiles that fit in SRAM. Same exact output as standard attention but 2-4x faster and uses O(n) memory instead of O(n²). The standard implementation in all modern frameworks. - **Multi-Query Attention (MQA)**: All heads share the same K and V projections. Reduces KV cache size by h× during inference, dramatically increasing batch size for serving. - **Grouped-Query Attention (GQA)**: Compromise between MHA and MQA — groups of heads share K/V. Used in LLaMA-2 70B, Mixtral, and most production LLMs. - **Sliding Window Attention**: Each token attends only to a local window of w neighboring tokens. O(n×w) complexity. Combined with global attention tokens (Longformer) or hierarchical structure for long-document processing. **Positional Information** Attention is permutation-equivariant — it has no notion of position. Positional encodings inject order information: - **Sinusoidal**: Fixed position-dependent sine/cosine patterns added to input embeddings. - **RoPE (Rotary Position Embedding)**: Applies position-dependent rotation to Q and K vectors before dot product. The relative position between two tokens is captured by the angle between their rotated vectors. The dominant approach for modern LLMs. Attention Mechanisms are **the computational primitive that replaced recurrence and convolution as the dominant method for modeling relationships in data** — a single, elegant operation that captures any dependency pattern the data requires, without the sequential bottleneck of RNNs or the fixed receptive field of CNNs.