cross-attention encoder-decoder

**Cross-Attention in Encoder-Decoder Models** is **the mechanism where decoder attends to encoder outputs to fuse input context during generation — enabling sequence-to-sequence tasks like translation, summarization, and visual question answering by dynamically selecting relevant input tokens at each decoding step**. **Encoder-Decoder Architecture Overview:** - **Dual Component**: encoder processes input sequence x=x₁...x_n → hidden states H_enc ∈ ℝ^(n×d); decoder generates output y=y₁...y_m with access to H_enc - **Information Flow**: encoder-decoder attention computes Attention(Q_dec, K_enc, V_enc) where Q comes from decoder, K,V from encoder outputs - **Self-Attention Layer**: decoder has own self-attention attending to previous decoder tokens y₁...y_i-₁ for causal generation - **Three-Layer Stack**: each decoder layer contains self-attention layer, cross-attention layer, and feed-forward layer sequentially **Cross-Attention Mechanism:** - **Query Source**: queries Q from current decoder hidden state h_dec_i ∈ ℝ^d at position i - **Key-Value Source**: keys K, values V from encoder output H_enc (reused across all decoder positions) - **Attention Scores**: computing α = softmax(Q·K_enc^T/√d_k) ∈ ℝ^(1×n) — probability distribution over n input tokens - **Context Vector**: c_i = Σ_j α_j · V_enc_j selecting weighted combination of encoder values — attended representation - **Output**: combining context with decoder state through linear projection — fused decoder representation **Mathematical Formulation:** - **Cross-Attention**: Q = h_dec·W_Q, K = H_enc·W_K, V = H_enc·W_V where W are learned projection matrices - **Scaled Dot Product**: Attention(Q,K,V) = softmax(QK^T/√d_k)V with scaling preventing gradient explosion - **Multi-Head**: splitting into h heads with dimension d_k = d/h — h=8 for base, h=16 for large models - **Concatenation**: outputs from h heads concatenated and projected: MultiHead = Concat(head₁,...,head_h)W_O **T5 Architecture Example:** - **Baseline Model**: 12-layer encoder, 12-layer decoder, 768 hidden dimension, 3072 FFN dimension — 220M parameters - **Attention Heads**: 12 heads in encoder self-attention, 12 heads in decoder cross-attention (full encoder output access) - **Layer Normalization**: post-LN architecture with layer norm before each sublayer (unusual convention) - **Performance**: T5-base achieves 61.5 ROUGE on CNN/DailyMail summarization, outperforming RoBERTa-based approaches **Cross-Attention Behavior and Properties:** - **Attention Pattern**: early layers focus on content words (nouns, verbs) while late layers focus on function words and structure - **Head Specialization**: different heads learn different alignment patterns — some focus on position-based, others on semantic alignment - **Entropy**: attention entropy typically 0.5-2.0 bits per position — fully peaked (entropy=0) on key tokens, diffuse on others - **Gradient Flow**: cross-attention gradients propagate back to encoder, enabling joint optimization of both components **Variants and Extensions:** - **Linear Cross-Attention**: replacing softmax with linear transformation QK^T (no normalization) — reduces complexity to O(n) for inference - **Sparse Cross-Attention**: restricting to top-k tokens or local window — enables attending to long input sequences (documents 10K+ tokens) - **Factorized Cross-Attention**: decomposing Q,K,V into low-rank components — reduces parameters and computation by 50-70% - **Hierarchical Cross-Attention**: using compressed encoder outputs (downsampled via pooling) — enables efficient long-context attention **Applications and Task-Specific Adaptations:** - **Machine Translation**: cross-attention learns input-output word alignment — supervised alignment signals (attention weights) interpretable - **Document Summarization**: attending to salient sentences and phrases — attention weights reveal which input contributes to each output token - **Visual Question Answering**: attending to image regions (spatial coordinates from CNN features) — cross-modal fusion of vision and language - **Code Generation**: attending to variable definitions in input context — enables referencing learned identifiers - **Abstractive QA**: attending to supporting evidence in document — improves factual grounding and citation accuracy **Inference and Computational Considerations:** - **Cache Reuse**: encoder outputs computed once and reused for all decoder steps — significant computation savings during generation - **Decoder-Only Decoding**: each decoder step processes decoder tokens (length 1 at step t) attending to full encoder (length n) — O(n) per step - **Batch Efficiency**: entire encoder batch processed together, decoders can interleave different sequence lengths — flexible batching - **Memory**: cross-attention KV cache stores full encoder features (n×d) vs growing decoder KV (t×d) — encoder dominates memory initially **Modern Alternatives and Comparisons:** - **Decoder-Only Models**: recent GPT-style models (GPT-3, Llama) use decoder-only with in-context examples instead of explicit encoder — simpler architecture - **Prefix Tuning**: conditioning decoder on frozen input representations — reduces tuning parameters to 0.1% while maintaining quality - **Adapter Modules**: injecting task-specific parameters in cross-attention layers — enables efficient multi-task learning - **Compressive Cross-Attention**: compressing encoder representations to memory vectors updated during training — reduces interference **Cross-Attention in Encoder-Decoder Models is fundamental to sequence-to-sequence learning — enabling dynamic information fusion from input context during generation across diverse tasks from translation to summarization to visual reasoning.**