attention mechanism deep learning

**Attention Mechanisms in Deep Learning** are **the neural network components that dynamically compute weighted combinations of input features based on learned relevance scores — enabling models to selectively focus on the most informative parts of the input, forming the foundation of Transformer architectures that dominate modern NLP, vision, and multimodal AI**. **Self-Attention (Scaled Dot-Product):** - **Query-Key-Value Framework**: input tokens projected into queries (Q), keys (K), and values (V) via learned linear transformations — attention output = softmax(QK^T/√d_k) × V where d_k is key dimension - **Scaling Factor**: division by √d_k prevents attention logits from growing with dimension — large logits push softmax into saturated regions with vanishing gradients; scaling maintains well-conditioned gradients - **Attention Matrix**: NxN matrix for sequence length N — each entry (i,j) represents how much token i attends to token j; quadratic memory and compute cost O(N²) limits maximum sequence length - **Softmax Normalization**: attention weights sum to 1 for each query position — creates a probability distribution over values; sharp weights (low temperature) focus on few tokens while uniform weights attend equally **Multi-Head Attention:** - **Parallel Heads**: h independent attention operations with separate Q, K, V projections — each head has dimension d_model/h, outputs concatenated and linearly projected back to d_model - **Specialization**: different heads learn different relationship patterns — some heads attend to syntactic (adjacent tokens), others to semantic (related meaning), positional (relative position), or hierarchical relationships - **Head Count**: typical choices: 8 heads (BERT-Base), 12 heads (BERT-Large), 32-128 heads (GPT-3/4) — more heads provide richer representation but diminishing returns beyond ~16 heads for most tasks - **Head Pruning**: many heads can be removed after training with minimal accuracy loss — structured pruning identifies and removes redundant heads for inference efficiency **Cross-Attention:** - **Encoder-Decoder Attention**: queries from decoder attend to keys and values from encoder output — enables the decoder to access source representation when generating target sequence (translation, summarization) - **Multimodal Attention**: queries from one modality attend to keys/values from another — image features attending to text features (or vice versa) in models like CLIP, Flamingo, and GPT-4V - **Memory Attention**: queries attend to external memory bank of key-value pairs — Retrieval-Augmented Generation (RAG) uses cross-attention to incorporate retrieved documents into generation **Attention mechanisms represent the most transformative innovation in deep learning since backpropagation — replacing the fixed-weight processing of traditional networks with dynamic, input-dependent computation that enables models to handle long-range dependencies, variable-length inputs, and cross-modal reasoning.**