gated fusion
**Gated Fusion** is a **multimodal fusion mechanism that learns dynamic, input-dependent weights for combining information from different modalities** — using sigmoid gating functions inspired by LSTM gates to automatically suppress noisy or uninformative modality channels and amplify reliable ones, enabling robust multimodal inference even when individual modalities degrade.
**What Is Gated Fusion?**
- **Definition**: A learned gating network produces scalar or vector weights that control how much each modality contributes to the fused representation, adapting per-sample rather than using fixed combination weights.
- **Gate Function**: z = σ(W_v·V + W_a·A + b), where σ is the sigmoid function, V and A are modality features, and z ∈ [0,1] controls the mixing ratio.
- **Fused Output**: h = z ⊙ V + (1−z) ⊙ A, where ⊙ is element-wise multiplication; when z→1 the model relies on vision, when z→0 it relies on audio.
- **Adaptive Behavior**: Unlike simple concatenation or averaging, gated fusion learns to ignore corrupted modalities — if audio is noisy, the gate automatically reduces its contribution.
**Why Gated Fusion Matters**
- **Robustness**: Real-world multimodal data often has missing or degraded modalities (occluded video, background noise); gated fusion gracefully handles these scenarios without manual intervention.
- **Efficiency**: Gating adds minimal parameters (one linear layer + sigmoid) compared to attention-based fusion, making it suitable for real-time and edge deployment.
- **Interpretability**: Gate values directly show which modality the model trusts for each input, providing built-in explainability for multimodal decisions.
- **Gradient Flow**: Sigmoid gates provide smooth gradients during backpropagation, enabling stable end-to-end training of the entire multimodal pipeline.
**Gated Fusion Variants**
- **Scalar Gating**: A single scalar z controls the global modality balance — simple but coarse, treating all feature dimensions equally.
- **Vector Gating**: A vector z ∈ R^d provides per-dimension control, allowing the model to trust different modalities for different feature aspects.
- **Multi-Gate Mixture of Experts (MMoE)**: Multiple gating networks route inputs to specialized expert sub-networks, extending gated fusion to multi-task multimodal learning.
- **Hierarchical Gating**: Gates at multiple network layers progressively refine the fusion, with early gates handling low-level feature selection and later gates controlling semantic-level combination.
| Fusion Method | Adaptivity | Parameters | Robustness | Interpretability |
|---------------|-----------|------------|------------|-----------------|
| Concatenation | None | 0 | Low | None |
| Averaging | None | 0 | Low | None |
| Scalar Gating | Per-sample | O(d) | Medium | High |
| Vector Gating | Per-sample, per-dim | O(d²) | High | High |
| Attention Fusion | Per-sample, per-token | O(d²) | High | Medium |
**Gated fusion is a lightweight yet powerful multimodal combination strategy** — learning input-dependent mixing weights that automatically suppress unreliable modalities and amplify informative ones, providing robust and interpretable multimodal inference with minimal computational overhead.