low-rank tensor fusion
**Low-Rank Tensor Fusion (LMF)** is an **efficient multimodal fusion method that approximates the full tensor outer product using low-rank decomposition** — reducing the computational complexity of tensor fusion from exponential to linear in the number of modalities while preserving the ability to model cross-modal interactions, making expressive multimodal fusion practical for real-time applications.
**What Is Low-Rank Tensor Fusion?**
- **Definition**: LMF approximates the weight tensor W of a multimodal fusion layer as a sum of R rank-1 tensors, where each rank-1 tensor is the outer product of modality-specific factor vectors, avoiding explicit computation of the full high-dimensional tensor.
- **Decomposition**: W ≈ Σ_{r=1}^{R} w_r^(1) ⊗ w_r^(2) ⊗ ... ⊗ w_r^(M), where w_r^(m) are learned factor vectors for each modality m and rank component r.
- **Efficient Computation**: Instead of computing the d₁×d₂×d₃ tensor explicitly, LMF computes R inner products per modality and combines them, reducing complexity from O(∏d_m) to O(R·Σd_m).
- **Origin**: Proposed by Liu et al. (2018) as a direct improvement over the Tensor Fusion Network, achieving comparable accuracy with orders of magnitude fewer parameters.
**Why Low-Rank Tensor Fusion Matters**
- **Scalability**: Full tensor fusion on three 256-dim modalities requires ~16.7M parameters; LMF with rank R=4 requires only ~3K parameters — a 5000× reduction enabling deployment on mobile and edge devices.
- **Speed**: Linear complexity in feature dimensions means LMF runs in milliseconds even for high-dimensional modality features, enabling real-time multimodal inference.
- **Preserved Expressiveness**: Despite the dramatic parameter reduction, LMF retains the ability to model cross-modal interactions because the low-rank factors span the most important interaction subspace.
- **End-to-End Training**: All factor vectors are jointly learned through backpropagation, automatically discovering the most informative cross-modal interaction patterns.
**How LMF Works**
- **Step 1 — Modality Encoding**: Each modality is encoded into a feature vector by its respective sub-network (CNN for images, LSTM/Transformer for text, spectrogram encoder for audio).
- **Step 2 — Factor Projection**: Each modality feature is projected through R learned factor vectors, producing R scalar values per modality.
- **Step 3 — Rank-1 Combination**: For each rank component r, the scalar projections from all modalities are multiplied together, capturing the cross-modal interaction for that component.
- **Step 4 — Summation**: The R rank-1 interaction values are summed and passed through a final classifier layer.
| Aspect | Full Tensor Fusion | Low-Rank (R=4) | Low-Rank (R=16) | Concatenation |
|--------|-------------------|----------------|-----------------|---------------|
| Parameters | O(∏d_m) | O(R·Σd_m) | O(R·Σd_m) | O(Σd_m) |
| Cross-Modal | All orders | Approximate | Better approx. | None |
| Memory | Very High | Very Low | Low | Very Low |
| Accuracy (MOSI) | 0.801 | 0.796 | 0.800 | 0.762 |
| Inference Speed | Slow | Fast | Fast | Fastest |
**Low-rank tensor fusion makes expressive multimodal interaction modeling practical** — decomposing the prohibitively large tensor outer product into a compact sum of rank-1 components that preserve cross-modal correlation capture while reducing parameters by orders of magnitude, enabling real-time multimodal AI on resource-constrained platforms.