joint space-time attention
**Joint space-time attention** is the **full attention strategy that computes interactions across all tokens from every frame simultaneously** - it provides maximal modeling flexibility but incurs very high memory and compute cost as clip length and resolution increase.
**What Is Joint Space-Time Attention?**
- **Definition**: Self-attention over flattened token set combining temporal and spatial dimensions in one sequence.
- **Token Count**: N equals T times H times W tokenized patches.
- **Complexity**: Quadratic in N, causing rapid scaling bottlenecks.
- **Primary Benefit**: Direct global coupling between any two space-time positions.
**Why Joint Attention Matters**
- **Expressive Power**: Captures arbitrary long-range dependencies without factorization constraints.
- **Modeling Simplicity**: One unified attention operation avoids hand-designed decomposition.
- **Research Baseline**: Useful upper-bound reference for architectural studies.
- **Short-Clip Strength**: Effective when token count is moderate.
- **Interpretability**: Full attention map shows direct global relation structure.
**Practical Limitations**
**Memory Explosion**:
- Attention matrix size grows with square of token count.
- Quickly becomes infeasible for long videos.
**Throughput Constraints**:
- Large quadratic operations reduce training and inference speed.
- Demands high-end hardware and careful batching.
**Scaling Workarounds**:
- Downsample input, shorten clips, or use sparse approximations.
- Often replaced by factorized or windowed attention designs.
**How It Works**
**Step 1**:
- Flatten all spatiotemporal tokens and project to queries, keys, and values.
**Step 2**:
- Compute full attention matrix, aggregate values globally, and pass through transformer blocks.
Joint space-time attention is **the most expressive but most expensive transformer video operation** - it is valuable as a conceptual gold standard and as a practical option only under constrained token budgets.