Recurrent Neural Networks Rnns
# Recurrent Neural Networks (RNNs)
## Introduction & Motivation
RNNs: process sequential data. LSTM, GRU, temporal dependencies. Applications: language modeling, time series prediction.
Motivation: Capture temporal patterns.
Applications: NLP, speech, video.
---
## Core Concepts & Theory
### Recurrent Computation
Hidden state updates.
### Long Short-Term Memory
Gated memory cells.
### Gated Recurrent Unit
Simplified gating.
### Bidirectional RNNs
Backward context.
---
## Mathematical Formulation
RNN Cell:
$$h_t = anh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$$
LSTM Cell:
$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$$
$$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$$
$$C_t = f_t \odot C_{t-1} + i_t \odot ilde{C}_t$$
GRU:
$$z_t = \sigma(W_z \cdot [h_{t-1}, x_t])$$
$$r_t = \sigma(W_r \cdot [h_{t-1}, x_t])$$
---
## Advanced Theory & Extensions
### Multi-Layer RNNs
Stacked recurrence.
### Attention in RNNs
Query-based focus.
### Sequence-to-Sequence
Encoder-decoder pairs.
---
## Computational Considerations
RNN forward: O(T·H²).
LSTM: O(T·H·(4H+I)).
Backpropagation: O(T·H²).
---
## Practical Implementation Strategies
### Gradient Clipping
Exploding gradient control.
### Initialization
Orthogonal initialization.
### Sequence Packing
Efficient batching.
---
## Benchmark Datasets & Evaluation
Penn Treebank: Language modeling.
MNIST Sequential: Pixel-by-pixel.
WikiText: Large-scale LM.
---
## Key Challenges & Limitations
### Vanishing Gradients
Long-term dependency issues.
### Computational Cost
Sequential processing.
### Memory Usage
Full sequence storage.
---
## Hyperparameter Tuning
Hidden size: 256-1024.
Dropout: 0.3-0.5.
Learning rate: 1e-4 to 1e-2.
---
## Real-World Applications & Case Studies
Machine Translation: Seq2seq models.
Language Modeling: Predictive text.
Time Series: Forecasting.
---
## Integration with Other Methods
RNNs + attention for focus; + embedding for language.
---
## Summary & Key Takeaways
RNNs capture temporal dependencies in sequences.
Principles:
1. Recurrent computation: Temporal processing.
2. LSTM: Long-range dependencies.
3. GRU: Simplified gating.
4. Bidirectional: Dual direction context.
5. Gradient control: Stability management.
---
## Appendix: Practical Labs
### Lab 1: RNN Cell Forward
import numpy as np
def rnn_cell_forward(x, h_prev, Wxh, Whh, by):
"""RNN cell forward pass"""
h = np.tanh(np.dot(Wxh, x) + np.dot(Whh, h_prev) + by)
return h
np.random.seed(42)
x = np.random.randn(10)
h_prev = np.random.randn(20)
Wxh = np.random.randn(20, 10) * 0.01
Whh = np.random.randn(20, 20) * 0.01
by = np.random.randn(20) * 0.01
h = rnn_cell_forward(x, h_prev, Wxh, Whh, by)
assert h.shape == h_prev.shape, "Correct hidden state shape"
print("✓ RNN cell forward working")### Lab 2: LSTM Cell
import numpy as np
def lstm_cell(x, h_prev, c_prev, Wf, Wi, Wo, Wc):
"""LSTM cell forward pass"""
combined = np.concatenate([h_prev, x])
f = 1 / (1 + np.exp(-Wf @ combined))
i = 1 / (1 + np.exp(-Wi @ combined))
o = 1 / (1 + np.exp(-Wo @ combined))
c_tilde = np.tanh(Wc @ combined)
c = f * c_prev + i * c_tilde
h = o * np.tanh(c)
return h, c
np.random.seed(42)
x = np.random.randn(10)
h_prev = np.random.randn(20)
c_prev = np.random.randn(20)
Wf = np.random.randn(20, 30) * 0.01
Wi = np.random.randn(20, 30) * 0.01
Wo = np.random.randn(20, 30) * 0.01
Wc = np.random.randn(20, 30) * 0.01
h, c = lstm_cell(x, h_prev, c_prev, Wf, Wi, Wo, Wc)
assert h.shape == h_prev.shape, "Correct LSTM output"
print("✓ LSTM cell working")### Lab 3: Gradient Clipping
import numpy as np
def clip_gradients(gradients, max_norm=1.0):
"""Clip RNN gradients by norm"""
grad_norm = np.sqrt(sum(np.sum(g**2) for g in gradients))
scale = min(1.0, max_norm / (grad_norm + 1e-8))
clipped = [g * scale for g in gradients]
return clipped
np.random.seed(42)
grads = [np.random.randn(10, 10) for _ in range(3)]
clipped = clip_gradients(grads, max_norm=1.0)
total_norm = np.sqrt(sum(np.sum(g**2) for g in clipped))
assert total_norm <= 1.01, "Gradients clipped"
print("✓ Gradient clipping working")### Lab 4: Sequence Packing
import numpy as np
def pack_sequences(sequences, max_length):
"""Pack variable-length sequences"""
packed = []
lengths = []
for seq in sequences:
if len(seq) <= max_length:
padded = np.pad(seq, (0, max_length - len(seq)))
else:
padded = seq[:max_length]
packed.append(padded)
lengths.append(min(len(seq), max_length))
return np.array(packed), np.array(lengths)
np.random.seed(42)
sequences = [np.random.randn(np.random.randint(5, 20)) for _ in range(8)]
packed, lengths = pack_sequences(sequences, max_length=20)
assert packed.shape[0] == 8, "Correct batch size"
assert packed.shape[1] == 20, "Correct sequence length"
print("✓ Sequence packing working")---