video understanding model
**Video Understanding Models** are **deep learning architectures designed to process and comprehend video data — modeling both spatial (per-frame visual content) and temporal (motion, causality, narrative) dimensions** — evolving from 3D CNNs and two-stream networks to video transformers and multimodal video-language models that can describe, answer questions about, and reason over video content.
**Architecture Evolution**
```
2D CNN + Pooling (early) → 3D CNN → Two-Stream → Video Transformers
→ Multimodal Video-Language Models (current)
```
**Key Architectures**
| Model | Type | Key Innovation |
|-------|------|---------------|
| C3D/I3D | 3D CNN | 3D convolutions over space+time |
| Two-Stream | Dual 2D CNN | Separate spatial (RGB) + temporal (optical flow) streams |
| SlowFast | Dual 3D CNN | Slow pathway (low FPS, rich spatial) + Fast (high FPS, temporal) |
| TimeSformer | ViT for video | Divided space-time attention |
| ViViT | ViT for video | Factorized/tubelet embedding variants |
| VideoMAE | Self-supervised | Masked autoencoder for video (90% masking!) |
| InternVideo | Foundation model | Multimodal pretraining on video-text pairs |
**Temporal Modeling Approaches**
```
1. Early Fusion: Stack T frames as input → single 3D network
+ Simple, captures fine-grained motion
- Computationally heavy (T× more tokens/voxels)
2. Late Fusion: Process frames independently → aggregate
+ Efficient (reuse image model), easy to scale
- Misses cross-frame interactions
3. Factorized: Spatial attention per frame → temporal attention across frames
+ Efficient (O(N·T + N·T) vs O(N·T)²)
- Approximation of full spatiotemporal attention
TimeSformer: Divided attention (space → time alternating)
ViViT Model 3: Spatial then temporal transformer
4. Token Compression: Sample sparse frames + merge tokens
+ Handles long videos (minutes to hours)
- May miss important moments
```
**VideoMAE: Self-Supervised Video Pretraining**
Masks 90-95% of video patches (much higher than image MAE's 75%) and reconstructs the missing patches. The extreme masking ratio works because video has massive temporal redundancy — neighboring frames share most content. The pretrained encoder learns strong spatiotemporal representations transferable to action recognition, video QA, and temporal grounding.
**Video-Language Models**
Modern video understanding is increasingly multimodal:
- **VideoChatGPT / Video-LLaVA**: Frame sampling → visual encoder → project to LLM token space → LLM generates text response about the video
- **Temporal grounding**: Locate specific moments in a video given a text query ('find when the person picks up the red cup')
- **Dense captioning**: Generate timestamped descriptions of video events
**Challenges**
- **Computational cost**: Video has 30× more data than images per second. A 1-minute video at 30fps = 1,800 frames → millions of tokens. Strategies: sparse sampling (1-4 fps), token merging, efficient attention.
- **Long-form video**: Understanding hour-long videos (movies, lectures) requires hierarchical approaches — summarize segments, then reason over summaries.
- **Temporal reasoning**: Models still struggle with fine-grained temporal understanding (before/after, causality, counting sequential actions).
**Video understanding has progressed from task-specific classification to general-purpose video reasoning** — driven by video foundation models pretrained on massive video-text datasets, achieving human-comparable performance on action recognition while pushing toward the harder challenges of long-form comprehension, temporal reasoning, and embodied video understanding for robotics.