Video Understanding Models are deep learning architectures designed to process and comprehend video data — modeling both spatial (per-frame visual content) and temporal (motion, causality, narrative) dimensions — evolving from 3D CNNs and two-stream networks to video transformers and multimodal video-language models that can describe, answer questions about, and reason over video content.

Architecture Evolution

2D CNN + Pooling (early) → 3D CNN → Two-Stream → Video Transformers
  → Multimodal Video-Language Models (current)

Key Architectures

Temporal Modeling Approaches

1. Early Fusion: Stack T frames as input → single 3D network
   + Simple, captures fine-grained motion
   - Computationally heavy (T× more tokens/voxels)

2. Late Fusion: Process frames independently → aggregate
   + Efficient (reuse image model), easy to scale
   - Misses cross-frame interactions

3. Factorized: Spatial attention per frame → temporal attention across frames
   + Efficient (O(N·T + N·T) vs O(N·T)²)
   - Approximation of full spatiotemporal attention
   TimeSformer: Divided attention (space → time alternating)
   ViViT Model 3: Spatial then temporal transformer

4. Token Compression: Sample sparse frames + merge tokens
   + Handles long videos (minutes to hours)
   - May miss important moments

VideoMAE: Self-Supervised Video Pretraining

Masks 90-95% of video patches (much higher than image MAE's 75%) and reconstructs the missing patches. The extreme masking ratio works because video has massive temporal redundancy — neighboring frames share most content. The pretrained encoder learns strong spatiotemporal representations transferable to action recognition, video QA, and temporal grounding.

Video-Language Models

Modern video understanding is increasingly multimodal:

• VideoChatGPT / Video-LLaVA: Frame sampling → visual encoder → project to LLM token space → LLM generates text response about the video
• Temporal grounding: Locate specific moments in a video given a text query ('find when the person picks up the red cup')
• Dense captioning: Generate timestamped descriptions of video events

Challenges

• Computational cost: Video has 30× more data than images per second. A 1-minute video at 30fps = 1,800 frames → millions of tokens. Strategies: sparse sampling (1-4 fps), token merging, efficient attention.
• Long-form video: Understanding hour-long videos (movies, lectures) requires hierarchical approaches — summarize segments, then reason over summaries.
• Temporal reasoning: Models still struggle with fine-grained temporal understanding (before/after, causality, counting sequential actions).

Video understanding has progressed from task-specific classification to general-purpose video reasoning — driven by video foundation models pretrained on massive video-text datasets, achieving human-comparable performance on action recognition while pushing toward the harder challenges of long-form comprehension, temporal reasoning, and embodied video understanding for robotics.