optical flow networks
**Optical flow networks** are the **deep models that estimate per-pixel motion vectors between frames to describe apparent displacement over time** - they provide foundational motion signals for tracking, action understanding, and video restoration pipelines.
**What Are Optical Flow Networks?**
- **Definition**: Neural architectures that predict dense 2D motion field from two or more frames.
- **Output Format**: For each pixel, horizontal and vertical displacement components.
- **Classical Assumption**: Brightness consistency plus spatial smoothness in local neighborhoods.
- **Modern Variants**: Encoder-decoder, pyramid warping, recurrent refinement, and transformer flow models.
**Why Optical Flow Matters**
- **Motion Primitive**: Core representation for temporal correspondence across frames.
- **Downstream Utility**: Improves detection, segmentation, frame interpolation, and stabilization.
- **Alignment Backbone**: Enables feature warping for multi-frame aggregation tasks.
- **Interpretability**: Flow vectors offer explicit motion visualization.
- **System Performance**: Good flow quality often directly lifts many video tasks.
**Flow Network Components**
**Feature Extraction**:
- Build robust descriptors for each frame.
- Multi-scale pyramids help large displacement handling.
**Matching or Correlation**:
- Compare features across frames to identify correspondences.
- Cost volumes encode candidate match quality.
**Refinement Head**:
- Iteratively update flow estimates to reduce residual error.
- Often includes smoothness regularization.
**How It Works**
**Step 1**:
- Encode frame pair into feature pyramids and compute matching cues with correlation or cost volume.
**Step 2**:
- Predict coarse flow and iteratively refine to final dense motion field.
Optical flow networks are **the motion-estimation engine that underpins correspondence-aware video intelligence** - strong flow prediction is a major multiplier for both understanding and generation tasks.