multimodal fusion
**Multimodal Fusion Strategies** are the **architectural approaches for combining information from multiple input modalities (text, image, audio, video, sensor data) into a unified representation** — ranging from simple concatenation to sophisticated cross-attention mechanisms, where the choice of when and how to fuse modalities critically determines model performance, with early fusion capturing low-level cross-modal interactions and late fusion preserving modality-specific processing before combining high-level decisions.
**Fusion Taxonomy**
| Strategy | When Fusion Occurs | How | Pros / Cons |
|----------|-------------------|-----|-------------|
| Early fusion | Input level | Concatenate raw inputs | Rich interaction / Hard to align |
| Mid fusion | Feature level | Cross-attention or concat features | Balanced / Complex |
| Late fusion | Decision level | Combine predictions | Simple / Misses interactions |
| Cross-attention | Throughout network | Attend across modalities | Powerful / Expensive |
| Bottleneck | Via shared tokens | Fusion tokens attend to all modalities | Efficient / Info bottleneck |
**Early Fusion**
```
[Image patches] + [Text tokens] → [Concatenated sequence]
↓
[Shared Transformer] → processes all tokens jointly
↓
[Output]
Example: VisualBERT, early multimodal transformers
Pros: Maximum interaction between modalities from layer 1
Cons: Need same architecture for both modalities, expensive
```
**Late Fusion**
```
[Image] → [Vision Encoder] → [Image embedding]
[Text] → [Text Encoder] → [Text embedding]
↓
[Concatenate / MLP / Voting]
↓
[Output]
Example: CLIP (dual encoder, late similarity)
Pros: Can use specialized encoders per modality
Cons: No deep cross-modal reasoning
```
**Cross-Attention Fusion**
```
[Image features] [Text features]
↓ ↓
Values/Keys Queries
↓ ↓
[Cross-Attention: Text queries attend to image features]
↓
[Fused representation]
Example: Flamingo, LLaVA, GPT-4V
Pros: Rich cross-modal reasoning — text can selectively focus on image regions
Cons: O(N_text × N_image) computation
```
**Bottleneck Fusion (Perceiver / Q-Former)**
```
[Image features: 1000+ tokens] [Text features]
↓ ↓
[Learned bottleneck queries: 32-64 tokens]
Queries cross-attend to image → compressed visual features
↓
[Fused with text via language model]
Example: BLIP-2 Q-Former, Perceiver
Pros: Compress high-dimensional modality, efficient
Cons: Information loss through bottleneck
```
**Fusion in Modern VLMs**
| Model | Fusion Strategy | Details |
|-------|----------------|--------|
| CLIP | Late (dual encoder) | Separate encoders, cosine similarity |
| LLaVA | Linear projection | Visual tokens projected into LLM input space |
| Flamingo | Cross-attention layers | Interleaved cross-attention in LLM |
| BLIP-2 | Bottleneck (Q-Former) | 32 queries bridge vision and language |
| GPT-4V / Gemini | Native early fusion | Multimodal tokens processed jointly |
**When to Use Which**
| Scenario | Best Strategy | Why |
|----------|-------------|-----|
| Retrieval (image↔text search) | Late fusion (CLIP-style) | Need separate embeddings |
| Visual QA | Cross-attention | Text must query specific image regions |
| Video + audio + text | Bottleneck | Compress high-dimensional modalities |
| Sensor fusion (self-driving) | Mid fusion | Need spatial alignment |
| Medical (image + clinical notes) | Cross-attention | Deep cross-modal reasoning |
**Challenges**
| Challenge | Why |
|-----------|-----|
| Modality imbalance | One modality dominates, others ignored |
| Missing modalities | What if audio is missing at test time? |
| Alignment | Spatial/temporal correspondence across modalities |
| Computational cost | Cross-attention scales quadratically |
Multimodal fusion is **the architectural challenge at the heart of building AI systems that perceive the world through multiple senses** — the choice between early, mid, late, or cross-attention fusion determines whether a model can perform deep cross-modal reasoning or only shallow comparison, making fusion strategy one of the most impactful design decisions in multimodal AI.