Optical Flow Motion Estimation
# Optical Flow & Motion Estimation
## Introduction & Motivation
Optical Flow: estimate pixel-level motion between frames. Dense motion fields; apparent motion. Applications: video compression, motion segmentation, autonomous driving.
Motivation: Capture dense motion information; interpret temporal changes.
Applications: Video analysis, motion tracking, scene understanding.
---
## Core Concepts & Theory
### Brightness Consistency
Pixels maintain intensity across frames.
### Smoothness Constraint
Neighboring pixels move similarly.
### Horn-Schunck Method
Variational approach with smoothness regularization.
### Lucas-Kanade Method
Local motion estimation via least squares.
---
## Mathematical Formulation
Optical Flow Constraint:
$$I_x u + I_y v + I_t = 0$$
Horn-Schunck Energy:
$$E = \int (I_x u + I_y v + I_t)^2 + \lambda(\|
abla u\|^2 + \|
abla v\|^2) dx$$
Lucas-Kanade:
$$\begin{bmatrix} u \\ v \end{bmatrix} = \left(\sum w_i I_x I_x^T
ight)^{-1} \sum w_i I_x I_t$$
---
## Advanced Theory & Extensions
### FlowNet
End-to-end CNN for optical flow.
### PWCNet
Pyramidal, warping, cost volume approach.
### RAFT
Recurrent all-pairs field transforms.
---
## Computational Considerations
Lucas-Kanade: O(N·window²).
Variational methods: O(N·iterations).
Deep learning: O(H·W·features).
---
## Practical Implementation Strategies
### Multi-Scale Processing
Coarse-to-fine flow estimation.
Warping: Use estimated flow to warp frames.
Iterative Refinement: Iteratively improve flow.
---
## Benchmark Datasets & Evaluation
Sintel: 1,064 frames, complex scenes.
KITTI: Real-world autonomous driving.
FlyingChairs: Synthetic large-scale dataset.
---
## Key Challenges & Limitations
### Occlusions
Pixels disappearing in next frame.
### Large Displacements
Motion exceeds neighborhood size.
### Untextured Regions
No visible motion gradient.
---
## Hyperparameter Tuning
Smoothness weight (λ): 0.01-1.0.
Window size (Lucas-Kanade): 7×7 to 15×15.
Pyramid levels: 4-6 levels.
---
## Real-World Applications & Case Studies
Video Stabilization: Remove camera shake via flow.
Object Tracking: Use flow for object motion.
Autonomous Driving: Estimate ego-motion.
---
## Integration with Other Methods
Optical flow + video understanding for action recognition; + segmentation for motion boundaries.
---
## Summary & Key Takeaways
Optical Flow via variational methods and deep learning enables dense motion field estimation.
Principles:
1. Brightness consistency: Core assumption.
2. Smoothness: Spatial coherence.
3. Variational formulation: Energy minimization.
4. Lucas-Kanade: Local estimation.
5. Deep learning: End-to-end learning.
---
---
## Appendix: Practical Labs
### Lab 1: Optical Flow Constraint
import numpy as np
def compute_flow_constraint(I_x, I_y, I_t):
"""Compute optical flow constraint residual"""
# I_x * u + I_y * v + I_t = 0
# For zero flow, residual = I_t
residual = I_t
return residual
# Test
np.random.seed(42)
I_x = np.random.randn(10, 10)
I_y = np.random.randn(10, 10)
I_t = np.random.randn(10, 10)
residual = compute_flow_constraint(I_x, I_y, I_t)
assert residual.shape == I_t.shape, "Residual shape"
print("✓ Flow constraint working")
if __name__ == "__main__":
print("Lab 1: FlowConstraint - PASSED")### Lab 2: Lucas-Kanade Motion
import numpy as np
def lucas_kanade_motion(I_x, I_y, I_t, window_size=5):
"""Lucas-Kanade local motion estimation"""
u = np.zeros_like(I_x, dtype=float)
v = np.zeros_like(I_y, dtype=float)
pad = window_size // 2
I_x_pad = np.pad(I_x, pad, mode='reflect')
I_y_pad = np.pad(I_y, pad, mode='reflect')
I_t_pad = np.pad(I_t, pad, mode='reflect')
for i in range(I_x.shape[0]):
for j in range(I_x.shape[1]):
# Extract window
I_x_win = I_x_pad[i:i+window_size, j:j+window_size].flatten()
I_y_win = I_y_pad[i:i+window_size, j:j+window_size].flatten()
I_t_win = I_t_pad[i:i+window_size, j:j+window_size].flatten()
# Build system
A = np.stack([I_x_win, I_y_win], axis=1)
b = -I_t_win
# Solve least squares
try:
flow = np.linalg.lstsq(A, b, rcond=None)[0]
u[i, j] = flow[0]
v[i, j] = flow[1]
except:
pass
return u, v
# Test
np.random.seed(42)
I_x = np.random.randn(10, 10)
I_y = np.random.randn(10, 10)
I_t = np.random.randn(10, 10)
u, v = lucas_kanade_motion(I_x, I_y, I_t)
assert u.shape == I_x.shape, "U shape"
assert v.shape == I_y.shape, "V shape"
print("✓ Lucas-Kanade working")
if __name__ == "__main__":
print("Lab 2: LucasKanade - PASSED")### Lab 3: Flow Magnitude
import numpy as np
def compute_flow_magnitude(u, v):
"""Compute magnitude of optical flow"""
magnitude = np.sqrt(u**2 + v**2)
return magnitude
# Test
np.random.seed(42)
u = np.random.randn(10, 10)
v = np.random.randn(10, 10)
mag = compute_flow_magnitude(u, v)
assert mag.shape == u.shape, "Magnitude shape"
assert np.all(mag >= 0), "Magnitude non-negative"
print("✓ Flow magnitude working")
if __name__ == "__main__":
print("Lab 3: FlowMagnitude - PASSED")### Lab 4: Warping with Flow
import numpy as np
def warp_frame(frame, u, v):
"""Warp frame using optical flow"""
h, w = frame.shape
y, x = np.meshgrid(np.arange(h), np.arange(w), indexing='ij')
# Compute target coordinates
x_new = x + u
y_new = y + v
# Clip to bounds
x_new = np.clip(x_new, 0, w - 1)
y_new = np.clip(y_new, 0, h - 1)
# Bilinear interpolation (simplified: nearest neighbor)
x_new = x_new.astype(int)
y_new = y_new.astype(int)
warped = frame[y_new, x_new]
return warped
# Test
np.random.seed(42)
frame = np.random.rand(10, 10)
u = np.random.randn(10, 10) * 0.5
v = np.random.randn(10, 10) * 0.5
warped = warp_frame(frame, u, v)
assert warped.shape == frame.shape, "Warped shape"
print("✓ Warping working")
if __name__ == "__main__":
print("Lab 4: Warping - PASSED")