cross-attention variants

**Cross-Attention Variants** are **modifications and extensions of the standard cross-attention mechanism** — where queries come from one sequence and keys/values from another, used for conditioning, fusion, and multimodal interaction. **Key Variants** - **Standard Cross-Attention**: Decoder queries attend to encoder keys/values (original Transformer). - **Perceiver Cross-Attention**: A small latent array cross-attends to a large input (bottleneck). - **Gated Cross-Attention**: Cross-attention output is gated before adding to the residual (Flamingo). - **Multi-Source**: Queries attend to multiple sources (e.g., text + image) with separate attention heads. - **Prompt Cross-Attention**: Attend to a set of learned prompt tokens (parameter-efficient tuning). **Why It Matters** - **Multimodal**: Cross-attention is the primary mechanism for fusing information across modalities (text-image, text-audio). - **Conditioning**: Used in diffusion models (Stable Diffusion) for text-conditioned image generation. - **Efficiency**: Perceiver-style cross-attention enables processing arbitrarily large inputs through a fixed-size bottleneck. **Cross-Attention Variants** are **the bridges between sequences** — the mechanism family that enables transformers to fuse, condition, and combine information across modalities.