tensor fusion
**Tensor Fusion** is a **multimodal fusion technique that captures all possible cross-modal interactions by computing the outer product of modality-specific feature vectors** — creating a high-dimensional tensor that explicitly encodes unimodal, bimodal, and trimodal feature interactions, enabling the model to discover complex inter-modal correlations that simpler fusion methods miss.
**What Is Tensor Fusion?**
- **Definition**: Given feature vectors from N modalities, tensor fusion computes their outer product to create an N-dimensional tensor containing every possible feature interaction across modalities.
- **Outer Product**: For vision V ∈ R^v, audio A ∈ R^a, and language L ∈ R^l, the fused tensor T = V ⊗ A ⊗ L ∈ R^(v×a×l) captures all v·a·l cross-modal interactions.
- **Augmented Vectors**: Each modality vector is augmented with a constant 1 (e.g., V' = [V; 1]) before the outer product, ensuring the tensor also contains unimodal and bimodal terms alongside trimodal interactions.
- **Tensor Fusion Network (TFN)**: The original architecture by Zadeh et al. (2017) that introduced this approach for multimodal sentiment analysis, achieving state-of-the-art results on CMU-MOSI and IEMOCAP benchmarks.
**Why Tensor Fusion Matters**
- **Complete Interaction Modeling**: Unlike concatenation (which only captures unimodal features) or bilinear fusion (which captures pairwise interactions), tensor fusion explicitly models all orders of cross-modal interaction in a single representation.
- **Expressiveness**: The outer product creates a feature space rich enough to represent subtle correlations — such as how a specific facial expression combined with a particular tone of voice and specific word choice indicates sarcasm.
- **Theoretical Foundation**: Tensor fusion provides a mathematically principled way to combine modalities, with connections to polynomial feature expansion and kernel methods.
- **Benchmark Performance**: TFN achieved significant improvements on multimodal sentiment analysis, emotion recognition, and speaker trait recognition tasks.
**Scalability Challenge and Solutions**
- **Dimensionality Explosion**: The outer product of three 256-dimensional vectors produces a 256³ ≈ 16.7 million dimensional tensor — computationally prohibitive for large feature dimensions.
- **Low-Rank Approximation (LMF)**: Decomposes the full tensor into a sum of R rank-1 tensors, reducing complexity from O(d^N) to O(R·N·d) while preserving most interaction information.
- **Factorized Multimodal Transformer**: Uses attention mechanisms to implicitly compute tensor interactions without materializing the full tensor.
- **Tucker Decomposition**: Represents the interaction tensor as a core tensor multiplied by factor matrices, providing a tunable compression ratio.
| Method | Complexity | Interactions Captured | Memory | Accuracy |
|--------|-----------|----------------------|--------|----------|
| Concatenation | O(Σd_i) | Unimodal only | Low | Baseline |
| Bilinear | O(d²) | Pairwise | Medium | Good |
| Full Tensor | O(∏d_i) | All orders | Very High | Best |
| Low-Rank Tensor | O(R·N·d) | Approximate all | Low | Near-best |
| Tucker Decomposition | O(R₁·R₂·R₃) | Compressed all | Medium | Good |
**Tensor fusion provides the most complete multimodal interaction modeling** — computing outer products across modality features to capture every possible cross-modal correlation, with low-rank approximations making this powerful approach practical for real-world multimodal AI systems.