Convolutional Recurrent Networks Convrnn
# Convolutional Recurrent Networks (ConvRNN)
## Introduction & Motivation
ConvRNN: combine convolution and recurrence. ConvLSTM, temporal convolution. Applications: video analysis, spatiotemporal modeling.
Motivation: Model spatial and temporal patterns jointly.
Applications: Video understanding, precipitation prediction.
---
## Core Concepts & Theory
### Convolutional LSTM
Spatial-temporal memory.
### 3D Convolution
Volumetric feature extraction.
### Temporal Convolution
Sequential convolution.
### Dilated Temporal Convolution
Enlarged temporal receptive fields.
---
## Mathematical Formulation
ConvLSTM Cell:
$$i_t = \sigma(W_{xi} * X_t + W_{hi} * H_{t-1} + b_i)$$
$$C_t = f_t \odot C_{t-1} + i_t \odot anh(W_{xc} * X_t + W_{hc} * H_{t-1} + b_c)$$
3D Convolution:
$$y[i,j,k] = \sum_m \sum_n \sum_p w[m,n,p] \cdot x[i+m, j+n, k+p]$$
---
## Advanced Theory & Extensions
### Multi-Layer ConvRNN
Stacked spatiotemporal.
### Residual Connections
Skip paths.
### Attention Mechanisms
Selective focus.
---
## Computational Considerations
ConvLSTM: O(T·H·W·C·K²).
3D Conv: O(T·H·W·C_in·C_out·K³).
Memory: O(T·H·W·C).
---
## Practical Implementation Strategies
### Padding Strategies
Boundary handling.
### Stride Selection
Temporal downsampling.
### Weight Sharing
Efficiency.
---
## Benchmark Datasets & Evaluation
Moving MNIST: Synthetic video.
Human3.6M: Action prediction.
UCF101: Action recognition.
---
## Key Challenges & Limitations
### Computational Cost
Heavy computation.
### Memory Usage
Sequence storage.
### Training Time
Long sequences.
---
## Hyperparameter Tuning
Temporal kernel: 3-5.
Spatial kernel: 3-7.
Stride: 1-2.
---
## Real-World Applications & Case Studies
Video Action Recognition: Temporal modeling.
Precipitation Prediction: Radar echo sequences.
Human Pose Prediction: Skeletal sequences.
---
## Integration with Other Methods
ConvRNN + attention for focus; + 3D CNN for feature learning.
---
## Summary & Key Takeaways
ConvRNN models spatiotemporal patterns efficiently.
Principles:
1. Convolutional LSTM: Spatial-temporal memory.
2. 3D convolution: Volumetric features.
3. Temporal convolution: Sequential processing.
4. Dilation: Receptive field expansion.
5. Efficiency: Parameter sharing.
---
## Appendix: Practical Labs
### Lab 1: ConvLSTM Cell
import numpy as np
def convlstm_cell(x, h_prev, c_prev, kernel_size=3):
"""ConvLSTM cell forward"""
combined = np.concatenate([x, h_prev], axis=-1)
f = 1 / (1 + np.exp(-np.mean(combined)))
i = 1 / (1 + np.exp(-np.mean(combined)))
o = 1 / (1 + np.exp(-np.mean(combined)))
c = f * c_prev + i * np.tanh(combined)
h = o * np.tanh(c)
return h, c
np.random.seed(42)
x = np.random.randn(32, 32, 64)
h = np.random.randn(32, 32, 64)
c = np.random.randn(32, 32, 64)
h_new, c_new = convlstm_cell(x, h, c)
assert h_new.shape == h.shape, "Correct output shape"
print("✓ ConvLSTM cell working")### Lab 2: 3D Convolution
import numpy as np
def convolve3d(x, w, stride=1):
"""3D convolution"""
d, h, w_size = w.shape
d_img, h_img, w_img = x.shape[:3]
d_out = (d_img - d) // stride + 1
h_out = (h_img - h) // stride + 1
w_out = (w_img - w_size) // stride + 1
out = np.zeros((d_out, h_out, w_out))
for i in range(d_out):
for j in range(h_out):
for k in range(w_out):
d_start = i * stride
h_start = j * stride
w_start = k * stride
patch = x[d_start:d_start+d, h_start:h_start+h,
w_start:w_start+w_size]
out[i, j, k] = np.sum(patch * w)
return out
np.random.seed(42)
x = np.random.randn(10, 28, 28)
w = np.random.randn(3, 3, 3)
out = convolve3d(x, w)
assert out.shape[0] <= x.shape[0], "Output depth reduced"
print("✓ 3D convolution working")### Lab 3: Temporal Downsampling
import numpy as np
def temporal_downsample(x, stride=2):
"""Downsample along temporal dimension"""
return x[::stride]
np.random.seed(42)
x = np.random.randn(30, 32, 32, 64)
downsampled = temporal_downsample(x, stride=2)
assert downsampled.shape[0] == 15, "Correct temporal downsampling"
print("✓ Temporal downsampling working")### Lab 4: Spatiotemporal Feature Extraction
import numpy as np
def extract_spatiotemporal_features(video, patch_size=4, temporal_stride=2):
"""Extract spatiotemporal patches"""
t, h, w, c = video.shape
patches = []
for i in range(0, t, temporal_stride):
if i + patch_size <= t:
temporal_patch = video[i:i+patch_size]
flat = temporal_patch.reshape(-1)
patches.append(flat)
return np.array(patches)
np.random.seed(42)
video = np.random.randn(30, 64, 64, 3)
features = extract_spatiotemporal_features(video)
assert features.shape[0] > 0, "Features extracted"
print("✓ Spatiotemporal feature extraction working")---