timesformer

**TimeSformer** is the **factorized video transformer that separates spatial attention and temporal attention to reduce computation while preserving long-range modeling** - by decomposing full 3D attention into two simpler steps, it scales better to longer clips and higher resolutions. **What Is TimeSformer?** - **Definition**: Transformer architecture applying attention across spatial tokens within each frame and across time for each spatial location. - **Factorized Scheme**: Spatial attention followed by temporal attention or alternative ordering. - **Token Format**: Frame patches converted to sequence with positional encodings. - **Design Goal**: Approximate full space-time reasoning at lower cost. **Why TimeSformer Matters** - **Compute Reduction**: Avoids quadratic blow-up of joint attention over all space-time tokens. - **Temporal Reach**: Can model longer clips with practical memory usage. - **Transformer Flexibility**: Retains global interactions along each axis. - **Benchmark Strength**: Competitive on action recognition datasets. - **Architecture Clarity**: Easy to reason about and ablate due to factorized blocks. **Factorization Variants** **Space Then Time**: - Capture per-frame visual structure first. - Then align dynamics across frames. **Time Then Space**: - Prioritize temporal traces at each patch location. - Then integrate per-frame context. **Divided Attention Blocks**: - Alternate temporal and spatial blocks through depth. - Improves receptive field with controlled complexity. **How It Works** **Step 1**: - Patchify frames into tokens and apply spatial attention in each frame. **Step 2**: - Reorganize tokens by spatial index and apply temporal attention across frames, then classify actions. TimeSformer is **a divide-and-conquer transformer design that delivers strong video modeling without full joint attention cost** - factorized attention makes long-clip processing significantly more practical.