3D Convolutional Networks

# 3D Convolutional Networks

## Introduction & Motivation

3D CNNs: process volumetric and video data. Spatiotemporal convolution. Applications: video understanding, medical imaging, action recognition.

Motivation: Learn spatiotemporal features jointly.

Applications: Video action recognition, 3D object detection.

---

## Core Concepts & Theory

### 3D Convolution

Three-dimensional feature extraction.

### Spatiotemporal Feature Maps

Combined spatial and temporal patterns.

### Inflated 2D Convolutions

Efficient 3D initialization.

### Dense Sampling

Frame sampling strategies.

---

## Mathematical Formulation

3D Convolution:
$$y[i,j,k] = \sum_d \sum_h \sum_w w[d,h,w] \cdot x[i+d, j+h, k+w]$$

Spatiotemporal Complexity:
$$ ext{Ops} = H \cdot W \cdot T \cdot C_{in} \cdot C_{out} \cdot K_h \cdot K_w \cdot K_t$$

Receptive Field Expansion:
$$RF_t = 1 + (K_t - 1) \cdot \prod S_i$$

---

## Advanced Theory & Extensions

### Residual 3D Networks

Skip connections for video.

### Efficient Sampling

Sparse spatiotemporal sampling.

### Two-Stream Networks

Separate RGB and optical flow.

---

## Computational Considerations

3D Conv: O(T·H·W·C·K³).

Memory: O(T·H·W·C).

Training: Expensive spatiotemporal processing.

---

## Practical Implementation Strategies

### Temporal Stride

Downsample temporal dimension.

### Optical Flow Preprocessing

Motion feature extraction.

### Data Augmentation

Temporal transformations.

---

## Benchmark Datasets & Evaluation

UCF101: Action recognition standard.

HMDB51: Human motion database.

Kinetics: Large-scale video dataset.

---

## Key Challenges & Limitations

### Computational Cost

High memory and computation.

### Data Requirements

Large labeled datasets needed.

### Temporal Alignment

Synchronizing action annotations.

---

## Hyperparameter Tuning

Temporal kernel: 3-5.

Temporal stride: 1-2.

Frame sampling: 8-32 frames.

---

## Real-World Applications & Case Studies

Action Recognition: Sports video analysis.

Medical Imaging: 3D CT/MRI analysis.

Video Surveillance: Activity detection.

---

## Integration with Other Methods

3D CNN + optical flow fusion; + attention for temporal focus.

---

## Summary & Key Takeaways

3D CNNs learn spatiotemporal representations for video and volumetric data.

Principles:
1. 3D convolution: Volumetric filtering.
2. Spatiotemporal: Joint space-time modeling.
3. Inflation: Efficient initialization.
4. Sampling: Frame-level decisions.
5. Two-stream: RGB and motion fusion.

---

## Appendix: Practical Labs

### Lab 1: 3D Convolution Operation

import numpy as np

def conv3d(x, w, stride=1):
 """Simple 3D convolution"""
 d, h, w_size = w.shape
 d_img, h_img, w_img = x.shape[:3]
 
 d_out = (d_img - d) // stride + 1
 h_out = (h_img - h) // stride + 1
 w_out = (w_img - w_size) // stride + 1
 
 out = np.zeros((d_out, h_out, w_out))
 
 for i in range(d_out):
 for j in range(h_out):
 for k in range(w_out):
 patch = x[i*stride:i*stride+d, j*stride:j*stride+h, k*stride:k*stride+w_size]
 out[i, j, k] = np.sum(patch * w)
 
 return out

np.random.seed(42)
x = np.random.randn(10, 28, 28)
w = np.random.randn(3, 3, 3)
out = conv3d(x, w)
assert out.shape[0] <= x.shape[0], "Depth reduced"
print("✓ 3D convolution working")

### Lab 2: Optical Flow Computation

import numpy as np

def compute_optical_flow_simple(frame1, frame2):
 """Simplified optical flow estimation"""
 # Compute gradient
 dx = frame2 - frame1
 
 # Normalize to [-1, 1]
 flow = np.clip(dx, -1, 1)
 return flow

np.random.seed(42)
frame1 = np.random.rand(32, 32)
frame2 = frame1 + 0.1 * np.random.randn(32, 32)
flow = compute_optical_flow_simple(frame1, frame2)
assert flow.shape == frame1.shape, "Correct flow shape"
print("✓ Optical flow computation working")

### Lab 3: Temporal Pooling

import numpy as np

def temporal_average_pooling(video, stride=2):
 """Average pooling along temporal dimension"""
 downsampled = np.mean(video[::stride], axis=0)
 return downsampled

np.random.seed(42)
video = np.random.randn(30, 64, 64, 3)
pooled = temporal_average_pooling(video, stride=2)
assert pooled.shape == (64, 64, 3), "Correct pooled shape"
print("✓ Temporal pooling working")

### Lab 4: Two-Stream Fusion

import numpy as np

def two_stream_fusion(rgb_features, flow_features, alpha=0.5):
 """Fuse RGB and optical flow features"""
 fused = alpha * rgb_features + (1 - alpha) * flow_features
 return fused

np.random.seed(42)
rgb = np.random.randn(32, 512)
flow = np.random.randn(32, 512)
fused = two_stream_fusion(rgb, flow, alpha=0.5)
assert fused.shape == rgb.shape, "Correct fused shape"
print("✓ Two-stream fusion working")

---