vivit
**ViViT** is the **video vision transformer family that tokenizes clips into spatiotemporal tubelets and applies transformer attention over space and time** - it extends ViT principles to video with multiple factorization options for efficiency.
**What Is ViViT?**
- **Definition**: Vision transformer architecture for video with tubelet embedding and temporal modeling modules.
- **Tokenization Strategy**: Tubelets capture local motion by grouping pixels across consecutive frames.
- **Model Variants**: Joint space-time attention or factorized spatial-then-temporal encoders.
- **Output Tasks**: Action recognition and video understanding benchmarks.
**Why ViViT Matters**
- **Transformer Transfer**: Brings strong image-transformer design into video domain.
- **Flexible Scaling**: Factorized variants support larger clips under memory limits.
- **Long-Range Modeling**: Better global temporal context than short-kernel 3D CNNs in many settings.
- **Research Influence**: Helped establish transformer-first direction for video.
- **Extensibility**: Compatible with self-supervised pretraining and multimodal fusion.
**ViViT Design Options**
**Joint Encoder**:
- Attend over all spatiotemporal tokens together.
- Strong but expensive for long clips.
**Factorized Encoder**:
- Apply spatial transformer then temporal transformer.
- Better efficiency with minimal quality loss in many tasks.
**Hybrid Heads**:
- Combine global pooled tokens with temporal heads.
- Useful for long-video adaptation.
**How It Works**
**Step 1**:
- Split clip into tubelets, project to embeddings, and add positional encodings for space and time.
**Step 2**:
- Process tokens with selected ViViT attention scheme and classify actions with final head.
ViViT is **a foundational video-transformer formulation that made tubelet tokenization and factorized attention mainstream** - it remains a key reference for modern transformer video architecture design.