Cross-Attention in Encoder-Decoder Models is the mechanism where decoder attends to encoder outputs to fuse input context during generation — enabling sequence-to-sequence tasks like translation, summarization, and visual question answering by dynamically selecting relevant input tokens at each decoding step.

Encoder-Decoder Architecture Overview:

• Dual Component: encoder processes input sequence x=x₁...x_n → hidden states H_enc ∈ ℝ^(n×d); decoder generates output y=y₁...y_m with access to H_enc
• Information Flow: encoder-decoder attention computes Attention(Q_dec, K_enc, V_enc) where Q comes from decoder, K,V from encoder outputs
• Self-Attention Layer: decoder has own self-attention attending to previous decoder tokens y₁...y_i-₁ for causal generation
• Three-Layer Stack: each decoder layer contains self-attention layer, cross-attention layer, and feed-forward layer sequentially

Cross-Attention Mechanism:

• Query Source: queries Q from current decoder hidden state h_dec_i ∈ ℝ^d at position i
• Key-Value Source: keys K, values V from encoder output H_enc (reused across all decoder positions)
• Attention Scores: computing α = softmax(Q·K_enc^T/√d_k) ∈ ℝ^(1×n) — probability distribution over n input tokens
• Context Vector: c_i = Σ_j α_j · V_enc_j selecting weighted combination of encoder values — attended representation
• Output: combining context with decoder state through linear projection — fused decoder representation

Mathematical Formulation:

• Cross-Attention: Q = h_dec·W_Q, K = H_enc·W_K, V = H_enc·W_V where W are learned projection matrices
• Scaled Dot Product: Attention(Q,K,V) = softmax(QK^T/√d_k)V with scaling preventing gradient explosion
• Multi-Head: splitting into h heads with dimension d_k = d/h — h=8 for base, h=16 for large models
• Concatenation: outputs from h heads concatenated and projected: MultiHead = Concat(head₁,...,head_h)W_O

T5 Architecture Example:

• Baseline Model: 12-layer encoder, 12-layer decoder, 768 hidden dimension, 3072 FFN dimension — 220M parameters
• Attention Heads: 12 heads in encoder self-attention, 12 heads in decoder cross-attention (full encoder output access)
• Layer Normalization: post-LN architecture with layer norm before each sublayer (unusual convention)
• Performance: T5-base achieves 61.5 ROUGE on CNN/DailyMail summarization, outperforming RoBERTa-based approaches

Cross-Attention Behavior and Properties:

• Attention Pattern: early layers focus on content words (nouns, verbs) while late layers focus on function words and structure
• Head Specialization: different heads learn different alignment patterns — some focus on position-based, others on semantic alignment
• Entropy: attention entropy typically 0.5-2.0 bits per position — fully peaked (entropy=0) on key tokens, diffuse on others
• Gradient Flow: cross-attention gradients propagate back to encoder, enabling joint optimization of both components

Variants and Extensions:

• Linear Cross-Attention: replacing softmax with linear transformation QK^T (no normalization) — reduces complexity to O(n) for inference
• Sparse Cross-Attention: restricting to top-k tokens or local window — enables attending to long input sequences (documents 10K+ tokens)
• Factorized Cross-Attention: decomposing Q,K,V into low-rank components — reduces parameters and computation by 50-70%
• Hierarchical Cross-Attention: using compressed encoder outputs (downsampled via pooling) — enables efficient long-context attention

Applications and Task-Specific Adaptations:

• Machine Translation: cross-attention learns input-output word alignment — supervised alignment signals (attention weights) interpretable
• Document Summarization: attending to salient sentences and phrases — attention weights reveal which input contributes to each output token
• Visual Question Answering: attending to image regions (spatial coordinates from CNN features) — cross-modal fusion of vision and language
• Code Generation: attending to variable definitions in input context — enables referencing learned identifiers
• Abstractive QA: attending to supporting evidence in document — improves factual grounding and citation accuracy

Inference and Computational Considerations:

• Cache Reuse: encoder outputs computed once and reused for all decoder steps — significant computation savings during generation
• Decoder-Only Decoding: each decoder step processes decoder tokens (length 1 at step t) attending to full encoder (length n) — O(n) per step
• Batch Efficiency: entire encoder batch processed together, decoders can interleave different sequence lengths — flexible batching
• Memory: cross-attention KV cache stores full encoder features (n×d) vs growing decoder KV (t×d) — encoder dominates memory initially

Modern Alternatives and Comparisons:

• Decoder-Only Models: recent GPT-style models (GPT-3, Llama) use decoder-only with in-context examples instead of explicit encoder — simpler architecture
• Prefix Tuning: conditioning decoder on frozen input representations — reduces tuning parameters to 0.1% while maintaining quality
• Adapter Modules: injecting task-specific parameters in cross-attention layers — enables efficient multi-task learning
• Compressive Cross-Attention: compressing encoder representations to memory vectors updated during training — reduces interference

Cross-Attention in Encoder-Decoder Models is fundamental to sequence-to-sequence learning — enabling dynamic information fusion from input context during generation across diverse tasks from translation to summarization to visual reasoning.