recurrent neural networks sequential processing temporal dependencies
# Recurrent Neural Networks: Sequential Processing & Temporal Dependencies
## Introduction & Motivation
Recurrent neural networks: process sequences; recurrent connections capture temporal dependencies. Vanilla RNN: simple but suffers vanishing gradients. LSTM/GRU: gating mechanisms; improved gradient flow. Bidirectional: process forward and backward; capture full context. Applications: language modeling, machine translation, time series, speech recognition.
Motivation: Sequences: dependencies over time. RNNs maintain hidden state; exploit temporal structure.
Applications: NLP, speech, time series.
---
## Core Concepts & Theory
### Hidden State
Maintains information across steps; h_t = f(x_t, h_{t-1}).
### Vanishing Gradients
Gradients → 0 over steps; shallow networks only.
### Gating Mechanisms
LSTM/GRU gates: input, forget, output; control information flow.
---
## Mathematical Formulation
Vanilla RNN:
$$h_t = anh(W_{xh} x_t + W_{hh} h_{t-1} + b_h)$$
$$y_t = W_{hy} h_t + b_y$$
LSTM cell:
$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$$
$$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$$
$$C_t = f_t \odot C_{t-1} + i_t \odot anh(W_C \cdot [h_{t-1}, x_t] + b_C)$$
---
## Advanced Theory & Extensions
### Attention Mechanisms
Attend to relevant steps; focus on important context.
### Bidirectional RNNs
Process forward and backward; use full sequence context.
### Hierarchical RNNs
Multi-level RNNs; different temporal scales.
---
## Computational Considerations
Vanilla RNN: O(T·h²) for T steps, h hidden dim; fast but unstable.
LSTM/GRU: O(4·T·h²) more gates; 4× vanilla.
Backprop through time: O(T·h) gradient computation.
---
## Practical Implementation Strategies
### Gradient Clipping
Prevent exploding gradients; clip by norm.
### Bidirectional Processing
Process forward and backward; concatenate hidden states.
### Residual Connections
Skip connections between layers; stabilize deep RNNs.
---
## Benchmark Datasets & Evaluation
Penn Treebank: Language modeling; perplexity metric.
WMT: Machine translation; BLEU score.
SQuAD: Reading comprehension; F1 score.
---
## Key Challenges & Limitations
### Vanishing Gradients
Deep sequences suffer; shallow only ~7-8 steps.
### Computational Cost
BPTT: backprop through time expensive for long sequences.
### Sequential Processing
Can't parallelize; unlike transformer attention.
---
## Hyperparameter Tuning
Hidden dimension: 64-512 typical; dataset dependent.
Number of layers: 1-3 standard; deeper needs residuals.
Bidirectional: On by default; improves performance.
---
## Real-World Applications & Case Studies
Machine Translation: LSTM-based seq2seq standard.
Language Modeling: LSTM/GRU; replaced by transformers.
Time Series: LSTM for forecasting; outperforms arima.
---
## Integration with Other Methods
RNN + Attention → seq2seq with attention.
RNN + Embedding → combine with representation learning.
---
## Summary & Key Takeaways
Recurrent neural networks via gating and bidirectional processing enable sequence modeling by maintaining temporal dependencies through recurrent connections.
Principles:
1. Hidden state: maintain information across steps.
2. Vanishing gradients: limits depth; LSTM/GRU mitigate.
3. Gating: control information flow; forget/input/output.
4. Bidirectional: forward + backward for full context.
5. Gradient clipping: essential for stability.
---
---
## Appendix: Practical Labs
### Lab 1: Vanilla RNN Cell
import torch
import torch.nn as nn
import numpy as np
class VanillaRNNCell(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.weight_ih = nn.Parameter(torch.randn(hidden_size, input_size))
self.weight_hh = nn.Parameter(torch.randn(hidden_size, hidden_size))
self.bias = nn.Parameter(torch.zeros(hidden_size))
def forward(self, x, h):
"""Single RNN step"""
h_new = torch.tanh(x @ self.weight_ih.T + h @ self.weight_hh.T + self.bias)
return h_new
# Test
np.random.seed(42)
cell = VanillaRNNCell(input_size=10, hidden_size=20)
x = torch.randn(32, 10)
h = torch.randn(32, 20)
h_new = cell(x, h)
assert h_new.shape == (32, 20), "Output shape correct"
assert torch.isfinite(h_new).all(), "All finite"
print("✓ Vanilla RNN working")
if __name__ == "__main__":
print("Lab 1: VanillaRNN - PASSED")### Lab 2: Sequence Processing
import torch
import numpy as np
def process_sequence(X_seq, rnn_cell, h_init):
"""Process entire sequence through RNN"""
T, batch_size, input_dim = X_seq.shape
hidden_dim = h_init.shape[1]
hidden_states = []
h = h_init
for t in range(T):
h = rnn_cell(X_seq[t], h)
hidden_states.append(h)
return torch.stack(hidden_states)
# Test
np.random.seed(42)
T, batch_size, input_dim = 10, 32, 20
X_seq = torch.randn(T, batch_size, input_dim)
# Simple RNN cell simulation
class SimpleRNN:
def __call__(self, x, h):
return torch.tanh(x + h)
rnn = SimpleRNN()
h_init = torch.randn(batch_size, input_dim)
outputs = process_sequence(X_seq, rnn, h_init)
assert outputs.shape == (T, batch_size, input_dim), "Output shape correct"
print("✓ Sequence processing working")
if __name__ == "__main__":
print("Lab 2: Sequence - PASSED")### Lab 3: Gradient Clipping
import torch
import numpy as np
def clip_grad_norm(model, max_norm=1.0):
"""Clip gradients by global norm"""
total_norm = 0
for p in model.parameters():
if p.grad is not None:
total_norm += p.grad.data.norm(2).item() ** 2
total_norm = total_norm ** 0.5
if total_norm > max_norm:
clip_factor = max_norm / total_norm
for p in model.parameters():
if p.grad is not None:
p.grad.data.mul_(clip_factor)
return total_norm
# Test
np.random.seed(42)
model = torch.nn.Linear(100, 50)
# Create large gradients
model.weight.grad = torch.randn_like(model.weight) * 10
norm = clip_grad_norm(model, max_norm=1.0)
assert norm > 0, "Norm should be positive"
print("✓ Gradient clipping working")
if __name__ == "__main__":
print("Lab 3: Clipping - PASSED")### Lab 4: Bidirectional RNN
import torch
import numpy as np
def bidirectional_process(X_seq, rnn_forward, rnn_backward, h_init):
"""Process sequence bidirectionally"""
T, batch_size, input_dim = X_seq.shape
# Forward
h_f = h_init
forward_states = []
for t in range(T):
h_f = rnn_forward(X_seq[t], h_f)
forward_states.append(h_f)
# Backward
h_b = h_init
backward_states = []
for t in range(T-1, -1, -1):
h_b = rnn_backward(X_seq[t], h_b)
backward_states.append(h_b)
backward_states.reverse()
# Concatenate
bidir = torch.cat([torch.stack(forward_states), torch.stack(backward_states)], dim=2)
return bidir
# Test
np.random.seed(42)
T, batch_size, input_dim = 10, 32, 20
X_seq = torch.randn(T, batch_size, input_dim)
class SimpleRNN:
def __call__(self, x, h):
return torch.tanh(x * 0.5 + h * 0.5)
rnn_f = SimpleRNN()
rnn_b = SimpleRNN()
h_init = torch.randn(batch_size, input_dim)
output = bidirectional_process(X_seq, rnn_f, rnn_b, h_init)
assert output.shape[2] == 2 * input_dim, "Should be bidirectional"
print("✓ Bidirectional RNN working")
if __name__ == "__main__":
print("Lab 4: Bidirectional - PASSED")