Structure from motion (SfM) for video is the geometric reconstruction process that jointly estimates camera poses and sparse 3D scene structure from feature correspondences across frames - it is a foundational method for building 3D maps from ordinary video.

What Is SfM?

• Definition: Recover scene geometry and camera trajectory by matching keypoints across multiple views.
• Input Requirement: Sufficient camera motion and textured features for reliable matching.
• Core Outputs: Camera extrinsics and sparse 3D point cloud.
• Typical Pipeline: Feature detection, matching, triangulation, and bundle adjustment.

Why SfM Matters

• Geometry Backbone: Provides initialization for dense reconstruction and neural rendering.
• Pose Estimation: Essential for AR, robotics, and mapping applications.
• No Depth Sensor Needed: Works with standard monocular video.
• Mature Tooling: Well-established algorithms and robust open-source implementations.
• Bridge Technology: Connects classical geometry and modern learned vision systems.

SfM Pipeline Stages

Feature Extraction and Matching:

• Detect repeatable keypoints and descriptors across frames.
• Build correspondence graph among views.

Incremental Reconstruction:

• Initialize from seed pair, triangulate points, and add cameras progressively.
• Maintain geometric consistency during expansion.

Bundle Adjustment:

• Optimize camera parameters and 3D points jointly.
• Reduce reprojection error globally.

How It Works

Step 1:

• Match features across video frames and estimate relative camera transforms.

Step 2:

• Triangulate 3D points and refine full reconstruction via bundle adjustment.

Structure from motion for video is the classical geometry engine that reconstructs scene structure and camera motion directly from image correspondences - it remains a critical first step in many advanced 3D video pipelines.