Video Understanding Temporal Modeling
# Video Understanding & Temporal Modeling
## Introduction & Motivation
Video Understanding: recognize actions and events in video sequences. Temporal modeling; motion patterns. Applications: action recognition, event detection, video classification.
Motivation: Capture temporal dynamics; learn from frame sequences.
Applications: Action recognition, video captioning, anomaly detection.
---
## Core Concepts & Theory
### 3D CNNs
Convolutional kernels over space-time.
### Two-Stream Networks
Separate spatial and temporal processing.
### Optical Flow
Compute motion between frames.
### Temporal Segmentation
Identify action boundaries.
---
## Mathematical Formulation
3D Convolution:
$$y_{ijkt} = \sum_{d_x,d_y,d_t} w_{d_x,d_y,d_t} \cdot x_{i+d_x,j+d_y,k+d_t}$$
Temporal Pooling:
$$ ext{pool}_t = \max_t f(t) ext{ or } ext{mean}_t f(t)$$
Two-Stream Fusion:
$$ ext{score} = \alpha \cdot ext{CNN}(I) + (1-\alpha) \cdot ext{CNN}(OF)$$
---
## Advanced Theory & Extensions
### I3D (Inflated 3D)
Inflate 2D networks to 3D.
### SlowFast
Dual-pathway temporal modeling.
### TSN (Temporal Segment Networks)
Sparse temporal sampling.
---
## Computational Considerations
3D convolution: O(T·H·W·C²).
Optical flow: O(H·W·T).
Temporal pooling: O(T·C).
---
## Practical Implementation Strategies
### Frame Sampling
Uniform, non-uniform, or random sampling.
### Temporal Augmentation
Speed variations, temporal dropout.
### Multi-Clip Testing
Ensemble predictions over multiple clips.
---
## Benchmark Datasets & Evaluation
UCF101: 13,320 videos, 101 action classes.
HMDB51: 6,766 videos, 51 action classes.
Kinetics-400: 306K videos, 400 action classes.
---
## Key Challenges & Limitations
### Computational Cost
High memory and computation for video.
### Variable Length Videos
Inconsistent temporal dimensions.
### Long-Range Dependencies
Capturing minute-level actions.
---
## Hyperparameter Tuning
Frame rate: 8-32 fps.
Clip length: 8-16 frames.
Temporal stride: 1-4 frames.
---
## Real-World Applications & Case Studies
Sports Analytics: Automatic highlight detection.
Surveillance: Suspicious activity detection.
Healthcare: Gesture recognition for accessibility.
---
## Integration with Other Methods
Video understanding + attention for frame weighting; + action localization for precise temporal boundaries.
---
## Summary & Key Takeaways
Video Understanding via 3D convolutions and temporal fusion enables action recognition from video sequences.
Principles:
1. 3D convolution: Space-time features.
2. Two-stream: Spatial and temporal paths.
3. Optical flow: Motion representation.
4. Temporal sampling: Efficient processing.
5. Multi-clip: Robust inference.
---
---
## Appendix: Practical Labs
### Lab 1: Temporal Pooling
import numpy as np
def temporal_pooling(video_features, pool_type='max'):
"""Pool features over temporal dimension"""
if pool_type == 'max':
pooled = np.max(video_features, axis=0)
elif pool_type == 'mean':
pooled = np.mean(video_features, axis=0)
elif pool_type == 'mean_max':
pooled = np.concatenate([np.mean(video_features, axis=0), np.max(video_features, axis=0)])
return pooled
# Test
np.random.seed(42)
features = np.random.randn(16, 512)
pooled_max = temporal_pooling(features, 'max')
pooled_mean = temporal_pooling(features, 'mean')
assert pooled_max.shape == (512,), "Max pooling shape"
assert pooled_mean.shape == (512,), "Mean pooling shape"
print("✓ Temporal pooling working")
if __name__ == "__main__":
print("Lab 1: TemporalPooling - PASSED")### Lab 2: Two-Stream Fusion
import numpy as np
def two_stream_fusion(spatial_scores, temporal_scores, alpha=0.5):
"""Fuse spatial and temporal streams"""
fused = alpha * spatial_scores + (1 - alpha) * temporal_scores
return fused
# Test
np.random.seed(42)
spatial = np.random.rand(101)
temporal = np.random.rand(101)
fused = two_stream_fusion(spatial, temporal)
assert fused.shape == spatial.shape, "Fused shape"
assert np.all((fused >= 0) & (fused <= 1)), "Fused in range"
print("✓ Two-stream fusion working")
if __name__ == "__main__":
print("Lab 2: TwoStreamFusion - PASSED")### Lab 3: Frame Sampling
import numpy as np
def sample_frames(num_frames, num_clips=3, sampling='uniform'):
"""Sample frame indices from video"""
if sampling == 'uniform':
indices = np.linspace(0, num_frames - 1, num_clips, dtype=int)
elif sampling == 'random':
indices = np.sort(np.random.choice(num_frames, num_clips, replace=False))
elif sampling == 'center':
center = num_frames // 2
offset = 5
indices = np.clip(center + np.arange(-offset, offset*2, offset*2//num_clips), 0, num_frames-1).astype(int)
return indices
# Test
num_frames = 100
indices = sample_frames(num_frames)
assert len(indices) == 3, "Correct number of samples"
assert np.all(indices < num_frames), "Valid indices"
print("✓ Frame sampling working")
if __name__ == "__main__":
print("Lab 3: FrameSampling - PASSED")### Lab 4: Video Classification
import numpy as np
def classify_video(clip_scores, method='majority_vote'):
"""Aggregate clip predictions to video label"""
if method == 'majority_vote':
predicted_labels = np.argmax(clip_scores, axis=1)
video_label = np.bincount(predicted_labels).argmax()
elif method == 'mean_score':
mean_scores = np.mean(clip_scores, axis=0)
video_label = np.argmax(mean_scores)
return video_label
# Test
np.random.seed(42)
clip_scores = np.random.rand(10, 101)
label_mv = classify_video(clip_scores, 'majority_vote')
label_ms = classify_video(clip_scores, 'mean_score')
assert 0 <= label_mv < 101, "Valid label (majority vote)"
assert 0 <= label_ms < 101, "Valid label (mean score)"
print("✓ Video classification working")
if __name__ == "__main__":
print("Lab 4: VideoClassification - PASSED")