divided space-time attention
**Divided Space-Time Attention** is a **computationally efficient factorization strategy for Video Vision Transformers that decomposes the prohibitively expensive Joint Space-Time Self-Attention operation into two sequential, independent attention stages — first applying Temporal Attention (each patch attends to itself across different video frames) and then applying Spatial Attention (each patch attends to its neighbors within the same frame) — drastically reducing computational complexity while preserving the model's ability to capture complex spatiotemporal dynamics.**
**The Joint Attention Catastrophe**
- **The Naive Approach**: A video clip of $T$ frames, each split into $N$ spatial patches, produces a total of $T imes N$ tokens. Joint Space-Time Attention computes the full $O((T imes N)^2)$ attention matrix. For a typical video input ($T = 8$ frames, $N = 196$ patches per frame), this produces a single attention matrix with $(1568)^2 = 2.46$ million entries per head — an enormous computational and memory burden that scales catastrophically with video length or resolution.
**The Divided Factorization**
TimeSformer (Facebook AI) proposed the elegant factorization into two sequential blocks per layer:
1. **Temporal Attention Block**: Each spatial patch at position $(x, y)$ attends exclusively to the same spatial position $(x, y)$ across all $T$ frames. This captures how a specific spatial location changes over time. The attention matrix is only $O(T^2)$ per position — if $T = 8$, this is a tiny $8 imes 8$ matrix per spatial position.
2. **Spatial Attention Block**: Each token at time $t$ attends exclusively to all $N$ spatial patches within the same frame at time $t$. This captures the spatial relationships between objects within a single snapshot. The attention matrix is $O(N^2)$ per frame.
**The Complexity Reduction**
- **Joint**: $O((T imes N)^2) = O(T^2 N^2)$
- **Divided**: $O(T^2 imes N + N^2 imes T) = O(TN(T + N))$
For $T = 8$, $N = 196$: Joint requires $sim 2.46M$ operations per head; Divided requires $sim 308K$ — an $8 imes$ reduction. As video length ($T$) or resolution ($N$) increases, the savings become even more dramatic.
**The Trade-Off**
Divided Attention assumes that spatiotemporal interactions can be adequately decomposed into separate spatial and temporal components. This is a reasonable approximation for most actions but can miss complex interactions where the spatial configuration of objects and their temporal dynamics are deeply entangled (e.g., a ball bouncing between two moving players requires simultaneous space-time reasoning).
**Divided Space-Time Attention** is **orthogonal dimensional processing** — treating Time and Space as independent, separable axes to simplify the overwhelming complexity of video reasoning into two tractable, sequential computations.