Recurrent Neural Networks LSTM GRU for Sequence Modeling
# Recurrent Neural Networks: LSTM & GRU for Sequence Modeling
## Introduction & Motivation
RNNs process sequential data by maintaining hidden state across time steps. LSTMs address vanishing gradients via memory cells and gates (input, forget, output). GRUs simplify with fewer parameters. Foundation for sequence-to-sequence, language modeling, time series prediction.
Motivation: Feedforward networks ignore sequential dependencies. RNNs share weights across time; LSTMs enable long-range dependencies via gating. Vanishing gradient problem solved.
Applications: Machine translation, speech recognition, sentiment analysis, time series forecasting, music generation.
---
## Core Concepts & Theory
### LSTM Cell
Memory cell \mathbf{c}_t and hidden state \mathbf{h}_t. Gates control information flow:
$$\mathbf{f}_t = \sigma(\mathbf{W}_f[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_f) \quad ext{forget gate}$$
$$\mathbf{i}_t = \sigma(\mathbf{W}_i[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_i) \quad ext{input gate}$$
$$ ilde{\mathbf{c}}_t = anh(\mathbf{W}_c[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_c) \quad ext{candidate}$$
$$\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot ilde{\mathbf{c}}_t$$
---
## Mathematical Formulation
Output gate and hidden state:
$$\mathbf{o}_t = \sigma(\mathbf{W}_o[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_o)$$
$$\mathbf{h}_t = \mathbf{o}_t \odot anh(\mathbf{c}_t)$$
GRU (simpler):
$$\mathbf{z}_t = \sigma(\mathbf{W}_z[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_z) \quad ext{reset gate}$$
$$\mathbf{r}_t = \sigma(\mathbf{W}_r[\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_r) \quad ext{update gate}$$
---
## Advanced Theory & Extensions
### Bidirectional RNNs
Forward and backward passes; access full sequence context.
### Attention Mechanism
Weight relevance of past states; focus on important steps.
### Gradient Flow
LSTM gates enable gradient flow via additive operations; mitigates vanishing gradients.
---
## Computational Considerations
Training: O(T imes n_h imes d) for T time steps, n_h hidden units, d features. BPTT (backprop through time).
Inference: O(T imes n_h) sequential (not parallelizable).
---
## Practical Implementation Strategies
### Hidden Unit Size
n_h = 64-512 typical; larger sequences or vocabulary → larger.
### Sequence Length
Truncate to fixed length; pad shorter sequences.
### Bidirectionality
Useful when full sequence available; not for streaming/generation.
---
## Benchmark Datasets & Evaluation
Text: Penn Treebank, WikiText. Metric: perplexity.
Time Series: Stock prices, weather. Metric: RMSE, MAE.
---
## Key Challenges & Limitations
### Computational Cost
Sequential processing; slow training vs. Transformers.
### Gradient Issues
Even LSTM suffers exploding gradients; use gradient clipping.
---
## Hyperparameter Tuning
hidden_size \in {64, 128, 256\}, num_layers \in {1, 2, 3\}, dropout \in {0.0, 0.3, 0.5\}.
---
## Real-World Applications & Case Studies
Google Translate: Seq2seq with attention.
Stock Prediction: LSTM for time series forecasting.
---
## Integration with Other Methods
LSTM + Attention → Transformer predecessor.
---
## Summary & Key Takeaways
LSTMs enable sequence modeling via gated memory cells, allowing long-range dependencies and gradient flow superior to vanilla RNNs.
Principles:
1. Gates control information flow.
2. Memory cells enable long-range dependencies.
3. LSTM preferred over GRU; GRU faster.
4. Bidirectional RNNs access full context.
5. Gradient clipping prevents explosion.
---
---
## Appendix: Practical Labs
### Lab 1: LSTM Classification
import torch
import torch.nn as nn
from sklearn.datasets import make_classification
import numpy as np
X, y = make_classification(n_samples=100, n_features=20, random_state=42)
X = X.reshape(100, 10, 2).astype('float32') # Reshape to (batch, seq_len, features)
class LSTMClassifier(nn.Module):
def __init__(self):
super().__init__()
self.lstm = nn.LSTM(input_size=2, hidden_size=32, batch_first=True)
self.fc = nn.Linear(32, 1)
def forward(self, x):
_, (h, c) = self.lstm(x)
return torch.sigmoid(self.fc(h[-1]))
model = LSTMClassifier()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.BCELoss()
X_t = torch.FloatTensor(X)
y_t = torch.FloatTensor(y).unsqueeze(1)
for _ in range(10):
opt.zero_grad()
pred = model(X_t)
loss = loss_fn(pred, y_t)
loss.backward()
opt.step()
print(f"Final loss: {loss.item():.4f}")
assert loss.item() < 1.0, "Loss should decrease"
print("✓ LSTM classification working")
if __name__ == "__main__":
print("Lab 1: LSTM Classification - PASSED")### Lab 2: GRU Sequence
import torch
import torch.nn as nn
seq_len, input_size, hidden_size, batch_size = 10, 5, 32, 8
X = torch.randn(batch_size, seq_len, input_size)
gru = nn.GRU(input_size=input_size, hidden_size=hidden_size, batch_first=True)
output, h_n = gru(X)
print(f"Output shape: {output.shape}, Hidden shape: {h_n.shape}")
assert output.shape == (batch_size, seq_len, hidden_size), "Output shape correct"
assert h_n.shape == (1, batch_size, hidden_size), "Hidden shape correct"
print("✓ GRU sequence working")
if __name__ == "__main__":
print("Lab 2: GRU Sequence - PASSED")### Lab 3: Bidirectional LSTM
import torch
import torch.nn as nn
seq_len, input_size, hidden_size, batch_size = 8, 10, 32, 4
X = torch.randn(batch_size, seq_len, input_size)
bi_lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size, bidirectional=True, batch_first=True)
output, (h_n, c_n) = bi_lstm(X)
print(f"Bidirectional output shape: {output.shape}")
assert output.shape == (batch_size, seq_len, hidden_size * 2), "Should have bidirectional output"
print("✓ Bidirectional LSTM working")
if __name__ == "__main__":
print("Lab 3: Bidirectional LSTM - PASSED")### Lab 4: Stacked LSTM
import torch
import torch.nn as nn
class StackedLSTM(nn.Module):
def __init__(self):
super().__init__()
self.lstm = nn.LSTM(input_size=10, hidden_size=32, num_layers=3, batch_first=True)
def forward(self, x):
output, (h, c) = self.lstm(x)
return output, h, c
model = StackedLSTM()
X = torch.randn(4, 8, 10) # (batch, seq_len, features)
output, h, c = model(X)
print(f"Output shape: {output.shape}, Hidden layers: {h.shape[0]}")
assert output.shape == (4, 8, 32), "Correct output shape"
assert h.shape[0] == 3, "Should have 3 layers"
print("✓ Stacked LSTM working")
if __name__ == "__main__":
print("Lab 4: Stacked LSTM - PASSED")