video understanding
**Video Understanding with Deep Learning** is the **application of neural networks to analyze, classify, and generate video content by modeling both spatial (within-frame) and temporal (across-frame) patterns** — extending image-based architectures with temporal reasoning capabilities to enable action recognition, video question answering, temporal grounding, and video generation, where the massive data volume (30 FPS × resolution × duration) creates unique computational challenges.
**Key Video Tasks**
| Task | Input | Output | Example |
|------|-------|--------|---------|
| Action Recognition | Video clip | Action class | "Playing basketball" |
| Temporal Action Detection | Untrimmed video | Action segments + labels | "Goal at 2:30-2:35" |
| Video Captioning | Video | Text description | "A dog catches a frisbee" |
| Video QA | Video + question | Answer | "What color is the car?" → "Red" |
| Video Generation | Text/image prompt | Video frames | Text→video synthesis |
| Video Object Tracking | Video + initial box | Object trajectory | Track person across frames |
**Architecture Evolution**
| Era | Architecture | Temporal Modeling |
|-----|------------|-------------------|
| 2014 | Two-Stream CNN | Optical flow + RGB, late fusion |
| 2017 | I3D (Inflated 3D) | 3D convolutions over space-time |
| 2019 | SlowFast | Dual pathways: slow (spatial) + fast (temporal) |
| 2021 | TimeSformer | Divided space-time attention |
| 2021 | ViViT | Video Vision Transformer |
| 2023 | VideoMAE v2 | Self-supervised pre-training |
| 2024+ | Video LLMs | LLM + visual encoder for video understanding |
**Temporal Modeling Strategies**
- **3D Convolution**: Extend 2D filters to 3D (H×W×T) → learn spatio-temporal features jointly.
- Computationally expensive: 3D conv ~ T× cost of 2D conv.
- **Temporal Attention**: Attend across frames at same spatial position.
- TimeSformer: Alternate spatial attention and temporal attention in separate blocks.
- **Frame Sampling**: Uniformly sample K frames (K=8-32) → process as image sequence.
- Efficient but may miss fast actions.
**SlowFast Networks**
- **Slow pathway**: Low frame rate (e.g., 4 FPS), high channel capacity → captures spatial semantics.
- **Fast pathway**: High frame rate (e.g., 32 FPS), low channel capacity → captures motion.
- Lateral connections fuse information between pathways.
- Key insight: Spatial semantics change slowly, motion information requires high temporal resolution.
**Video Foundation Models**
| Model | Type | Capability |
|-------|------|------------|
| InternVideo2 | Encoder | Action recognition, retrieval, captioning |
| VideoLLaMA | LLM-based | Video QA, conversation about videos |
| Sora | Generation | Text-to-video, minute-long coherent videos |
| Runway Gen-3 | Generation | High-quality short video generation |
**Challenges**
- **Computation**: Video data is 30-100x larger than images → memory and compute intensive.
- **Temporal reasoning**: Understanding causality, long-range temporal dependencies.
- **Long videos**: Hours of content → cannot process all frames → need intelligent sampling.
Video understanding is **one of the most active frontiers in deep learning** — the combination of spatial and temporal reasoning required for video pushes model architectures and compute requirements beyond what image understanding demands, with video generation (Sora-class models) representing the next major milestone in generative AI.