Action Recognition Video Classification 3D CNN
# Action Recognition: Video Classification & 3D CNN
## Introduction & Motivation
Action recognition: classify video/temporal data. 3D CNN: convolve spatially and temporally. Two-stream networks: spatial + optical flow. Applications: video understanding, action detection, behavior analysis.
Motivation: Videos contain temporal information. 2D CNNs insufficient; temporal modeling needed.
Applications: Video understanding, action detection, surveillance.
---
## Core Concepts & Theory
### 3D Convolution
Convolve over space and time; temporal feature.
### Temporal Pooling
Aggregate over time; reduce temporal dimension.
### Two-Stream Architecture
RGB + optical flow; complementary streams.
---
## Mathematical Formulation
3D convolution:
$$y[i,j,t] = \sum_{di,dj,dt} w[di,dj,dt] \cdot x[i+di, j+dj, t+dt]$$
Two-stream fusion:
$$y = \alpha \cdot ext{stream}_{ ext{spatial}} + (1-\alpha) \cdot ext{stream}_{ ext{temporal}}$$
---
## Advanced Theory & Extensions
### Temporal Segment Networks
Sparse sampling; temporal division.
### SlowFast Networks
Two pathways: slow (spatial) + fast (temporal).
### Attention over Time
Temporal attention; importance weighting.
---
## Computational Considerations
3D CNN: O(T·H·W·C·D) temporal added; expensive.
Two-Stream: O(optical_flow + RGB) parallel.
SlowFast: O(slow_path + fast_path) two pathways.
---
## Practical Implementation Strategies
### Optical Flow Computation
TVL1, FlowNet; temporal difference.
### Temporal Sampling
Uniform vs. dense; tradeoff coverage-computation.
### Data Augmentation
Temporal jittering, spatial augmentation.
---
## Benchmark Datasets & Evaluation
UCF101: Action recognition standard; 101 classes.
Kinetics: Large-scale video dataset; 400+ classes.
ActivityNet: Long-term action detection.
---
## Key Challenges & Limitations
### Temporal Redundancy
Lots of repeated frames; sparse sampling.
### Optical Flow
Expensive to compute; FastFlowNet.
### Real-Time Inference
Video processing expensive; model compression.
---
## Hyperparameter Tuning
Frame sampling rate: 2-4 frames per clip.
Clip length: 8-32 frames typical.
Temporal kernel: 3x3x3 standard; 3×3 spatial, 3 temporal.
---
## Real-World Applications & Case Studies
Video Understanding: YouTube action recognition.
Surveillance: Suspicious action detection.
Sports Analytics: Play classification.
---
## Integration with Other Methods
Action Recognition + Localization → spatio-temporal localization.
Action Recognition + Tracking → temporal tracking.
---
## Summary & Key Takeaways
Action recognition via 3D CNN and two-stream networks enables temporal modeling through spatial-temporal convolution and complementary feature streams.
Principles:
1. 3D convolution: temporal feature.
2. Two-stream: RGB + optical flow.
3. Temporal pooling: aggregation.
4. Optical flow: motion cues.
5. SlowFast: dual pathways.
---
---
## Appendix: Practical Labs
### Lab 1: 3D Convolution
import numpy as np
def conv3d_single_output(video, kernel, stride=1):
"""Single 3D convolution output"""
T_kernel, H_kernel, W_kernel, C_in, C_out = kernel.shape
T, H, W, C = video.shape
output_value = 0
for dt in range(T_kernel):
for dh in range(H_kernel):
for dw in range(W_kernel):
for c_in in range(C_in):
output_value += kernel[dt, dh, dw, c_in, 0] * video[dt, dh, dw, c_in]
return output_value
# Test
np.random.seed(42)
video = np.random.randn(8, 32, 32, 3) # T, H, W, C
kernel = np.random.randn(3, 3, 3, 3, 1) # T, H, W, C_in, C_out
output = conv3d_single_output(video, kernel)
assert np.isfinite(output), "Output finite"
print("✓ 3D convolution working")
if __name__ == "__main__":
print("Lab 1: Conv3D - PASSED")### Lab 2: Temporal Pooling
import numpy as np
def temporal_average_pooling(video_features, pool_size=2):
"""Average pooling over time"""
T, H, W, C = video_features.shape
T_out = T // pool_size
output = np.zeros((T_out, H, W, C))
for t in range(T_out):
start = t * pool_size
end = start + pool_size
output[t] = video_features[start:end].mean(axis=0)
return output
# Test
np.random.seed(42)
features = np.random.randn(8, 16, 16, 64)
pooled = temporal_average_pooling(features, pool_size=2)
assert pooled.shape == (4, 16, 16, 64), "Pooled shape"
print("✓ Temporal pooling working")
if __name__ == "__main__":
print("Lab 2: TemporalPooling - PASSED")### Lab 3: Two-Stream Fusion
import numpy as np
def two_stream_fusion(spatial_features, temporal_features, alpha=0.5):
"""Fuse spatial (RGB) and temporal (optical flow) streams"""
# Normalize
spatial_norm = spatial_features / (np.linalg.norm(spatial_features) + 1e-8)
temporal_norm = temporal_features / (np.linalg.norm(temporal_features) + 1e-8)
# Weighted fusion
fused = alpha * spatial_norm + (1 - alpha) * temporal_norm
return fused
# Test
np.random.seed(42)
spatial = np.random.randn(1, 1024)
temporal = np.random.randn(1, 1024)
fused = two_stream_fusion(spatial, temporal, alpha=0.6)
assert fused.shape == spatial.shape, "Fused shape"
print("✓ Two-stream fusion working")
if __name__ == "__main__":
print("Lab 3: TwoStreamFusion - PASSED")### Lab 4: Video Classification Metrics
import numpy as np
def top_k_accuracy(predictions, targets, k=5):
"""Compute top-k accuracy"""
# predictions: (N, C) logits
# targets: (N,) class indices
top_k_preds = np.argsort(-predictions, axis=1)[:, :k]
correct = np.any(top_k_preds == targets[:, np.newaxis], axis=1)
accuracy = correct.mean()
return accuracy
# Test
np.random.seed(42)
predictions = np.random.randn(100, 101) # 100 videos, 101 classes
targets = np.random.randint(0, 101, 100)
top1 = top_k_accuracy(predictions, targets, k=1)
top5 = top_k_accuracy(predictions, targets, k=5)
assert 0 <= top1 <= 1, "Top-1 accuracy in [0,1]"
assert 0 <= top5 <= 1, "Top-5 accuracy in [0,1]"
assert top5 >= top1, "Top-5 >= Top-1"
print("✓ Video classification metrics working")
if __name__ == "__main__":
print("Lab 4: VideoMetrics - PASSED")