Divided Space-Time Attention is a computationally efficient factorization strategy for Video Vision Transformers that decomposes the prohibitively expensive Joint Space-Time Self-Attention operation into two sequential, independent attention stages — first applying Temporal Attention (each patch attends to itself across different video frames) and then applying Spatial Attention (each patch attends to its neighbors within the same frame) — drastically reducing computational complexity while preserving the model's ability to capture complex spatiotemporal dynamics.

The Joint Attention Catastrophe

• The Naive Approach: A video clip of $T$ frames, each split into $N$ spatial patches, produces a total of $T imes N$ tokens. Joint Space-Time Attention computes the full $O((T imes N)^2)$ attention matrix. For a typical video input ($T = 8$ frames, $N = 196$ patches per frame), this produces a single attention matrix with $(1568)^2 = 2.46$ million entries per head — an enormous computational and memory burden that scales catastrophically with video length or resolution.

The Divided Factorization

TimeSformer (Facebook AI) proposed the elegant factorization into two sequential blocks per layer:

1. Temporal Attention Block: Each spatial patch at position $(x, y)$ attends exclusively to the same spatial position $(x, y)$ across all $T$ frames. This captures how a specific spatial location changes over time. The attention matrix is only $O(T^2)$ per position — if $T = 8$, this is a tiny $8 imes 8$ matrix per spatial position.

2. Spatial Attention Block: Each token at time $t$ attends exclusively to all $N$ spatial patches within the same frame at time $t$. This captures the spatial relationships between objects within a single snapshot. The attention matrix is $O(N^2)$ per frame.

The Complexity Reduction

• Joint: $O((T imes N)^2) = O(T^2 N^2)$
• Divided: $O(T^2 imes N + N^2 imes T) = O(TN(T + N))$

For $T = 8$, $N = 196$: Joint requires $sim 2.46M$ operations per head; Divided requires $sim 308K$ — an $8 imes$ reduction. As video length ($T$) or resolution ($N$) increases, the savings become even more dramatic.

The Trade-Off

Divided Attention assumes that spatiotemporal interactions can be adequately decomposed into separate spatial and temporal components. This is a reasonable approximation for most actions but can miss complex interactions where the spatial configuration of objects and their temporal dynamics are deeply entangled (e.g., a ball bouncing between two moving players requires simultaneous space-time reasoning).

Divided Space-Time Attention is orthogonal dimensional processing — treating Time and Space as independent, separable axes to simplify the overwhelming complexity of video reasoning into two tractable, sequential computations.