LSTM Recurrent Networks Sequential Modeling
# LSTM & Recurrent Networks: Sequential Modeling
## Introduction & Motivation
Long Short-Term Memory (LSTM): gated recurrent unit; addresses vanishing gradient. Cell state: long-term memory. Hidden state: short-term. Three gates: input, forget, output. Applications: sequence prediction, language modeling, time series.
Motivation: RNNs gradient vanish over long sequences. LSTM maintains gradient via multiplicative gates.
Applications: Speech recognition, machine translation, time series forecasting, video analysis.
---
## Core Concepts & Theory
### Cell State
Distinct from hidden state; carries long-term dependencies.
### Gates (Input, Forget, Output)
Learned sigmoid functions; control information flow.
### Peephole Connections
Gates attend to cell state (optional); improves performance.
---
## Mathematical Formulation
LSTM equations:
$$f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$$ (Forget gate)
$$i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$$ (Input gate)
$$ ilde{C}_t = anh(W_C \cdot [h_{t-1}, x_t] + b_C)$$ (Candidate)
$$C_t = f_t * C_{t-1} + i_t * ilde{C}_t$$ (Cell state)
$$o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$$ (Output gate)
$$h_t = o_t * anh(C_t)$$
---
## Advanced Theory & Extensions
### Bidirectional LSTM (BiLSTM)
Process sequence forward and backward; aggregate.
### Attention over LSTM
Attend to encoder hidden states; seq2seq.
### Coupled Input-Forget Gates (CIFG)
Simplification; reduces parameters.
---
## Computational Considerations
Per-timestep: O(D²) for matrix multiplies (4 gates).
Sequence: O(T × D²) for sequence length T.
Memory: O(T × D) to store hidden states.
---
## Practical Implementation Strategies
### Dropout on Recurrent Connections
Variational dropout: same mask per timestep.
### Gradient Clipping
Prevent gradient explosion; typical norm 5-20.
### Teacher Forcing
During training, use ground truth; inference uses predictions.
---
## Benchmark Datasets & Evaluation
Penn Treebank (Language Modeling): PPL < 80.
MNIST Sequential: 99% accuracy (sequential pixel input).
Time Series (Electricity): RMSE benchmarks.
---
## Key Challenges & Limitations
### Training Dynamics
Still suffer from vanishing gradients (less severe than vanilla RNN).
### Computational Cost
Slower than Transformers; not parallelizable.
### Truncated BPTT
Truncate backprop; limited long-range learning.
---
## Hyperparameter Tuning
Hidden size: 128-512; balance capacity-speed.
Number of layers: 1-3; deeper often unstable.
Dropout: 0.2-0.5 on recurrent connections.
---
## Real-World Applications & Case Studies
Speech Recognition: Acoustic modeling with BiLSTM.
Machine Translation: Encoder-decoder with attention.
Time Series: Stock price, weather forecasting.
---
## Integration with Other Methods
LSTM + Attention → Seq2Seq with attention.
LSTM + CRF → Sequence tagging with structured output.
---
## Summary & Key Takeaways
LSTMs via gated mechanisms enable long-range sequential learning by maintaining separate cell state, addressing gradient flow challenges in vanilla RNNs.
Principles:
1. Cell state distinct from hidden; long-term memory.
2. Forget gate: selectively reset past information.
3. Input gate: control new information flow.
4. Output gate: control hidden state exposure.
5. Gradient flow improved via additive state updates.
---
---
## Appendix: Practical Labs
### Lab 1: LSTM Cell
import torch
import torch.nn as nn
import numpy as np
class LSTMCell(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.input_size = input_size
self.hidden_size = hidden_size
# Gates: input, forget, cell, output
self.W = nn.Linear(input_size + hidden_size, 4 * hidden_size)
def forward(self, x, state):
h, c = state
combined = torch.cat([x, h], dim=1)
gates = self.W(combined)
i, f, g, o = gates.chunk(4, dim=1)
i = torch.sigmoid(i)
f = torch.sigmoid(f)
g = torch.tanh(g)
o = torch.sigmoid(o)
c_new = f * c + i * g
h_new = o * torch.tanh(c_new)
return h_new, c_new
# Test
cell = LSTMCell(input_size=10, hidden_size=20)
x = torch.randn(5, 10)
h = torch.zeros(5, 20)
c = torch.zeros(5, 20)
h_new, c_new = cell(x, (h, c))
print(f"New hidden shape: {h_new.shape}, Cell state shape: {c_new.shape}")
assert h_new.shape == (5, 20), "Should match hidden size"
assert c_new.shape == (5, 20), "Should match hidden size"
print("✓ LSTM cell working")
if __name__ == "__main__":
print("Lab 1: Cell - PASSED")### Lab 2: Sequence Processing
import torch
import torch.nn as nn
def process_sequence_lstm(x, hidden_size):
"""Process sequence with LSTM"""
# x: [T, B, D] (time, batch, input_dim)
T, B, D = x.shape
lstm = nn.LSTM(input_size=D, hidden_size=hidden_size, batch_first=False)
output, (h, c) = lstm(x)
return output, h, c
# Test
seq_len, batch_size, input_dim = 20, 8, 10
x = torch.randn(seq_len, batch_size, input_dim)
output, h, c = process_sequence_lstm(x, hidden_size=32)
print(f"Output shape: {output.shape}, Hidden shape: {h.shape}")
assert output.shape == (seq_len, batch_size, 32), "Should have hidden_size outputs"
assert h.shape == (1, batch_size, 32), "Hidden state shape"
print("✓ Sequence processing working")
if __name__ == "__main__":
print("Lab 2: Sequence - PASSED")### Lab 3: BiLSTM
import torch
import torch.nn as nn
class BiLSTM(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.lstm_fwd = nn.LSTM(input_size, hidden_size, batch_first=True)
self.lstm_bwd = nn.LSTM(input_size, hidden_size, batch_first=True)
def forward(self, x):
# Forward
out_fwd, _ = self.lstm_fwd(x)
# Backward (reverse sequence)
x_rev = torch.flip(x, [1])
out_bwd, _ = self.lstm_bwd(x_rev)
out_bwd = torch.flip(out_bwd, [1])
# Concatenate
output = torch.cat([out_fwd, out_bwd], dim=-1)
return output
# Test
bilstm = BiLSTM(input_size=10, hidden_size=20)
x = torch.randn(8, 15, 10) # [B, T, D]
output = bilstm(x)
print(f"BiLSTM output shape: {output.shape}")
assert output.shape == (8, 15, 40), "Should concatenate forward and backward"
print("✓ BiLSTM working")
if __name__ == "__main__":
print("Lab 3: BiLSTM - PASSED")### Lab 4: Gradient Flow
import torch
import torch.nn as nn
import numpy as np
def check_gradient_flow(x, seq_len):
"""Check gradient flow through LSTM"""
lstm = nn.LSTM(input_size=10, hidden_size=20, num_layers=2)
x = torch.randn(seq_len, x, 10, requires_grad=True)
output, _ = lstm(x)
loss = output.sum()
loss.backward()
# Check gradient magnitudes at different layers
first_layer_grad = x.grad.abs().mean()
return first_layer_grad.item()
# Test
grad_mag = check_gradient_flow(8, seq_len=50)
print(f"Gradient magnitude: {grad_mag:.4f}")
assert grad_mag > 0, "Should have non-zero gradients"
assert np.isfinite(grad_mag), "Gradients should not explode/vanish"
print("✓ Gradient flow checking working")
if __name__ == "__main__":
print("Lab 4: Gradient Flow - PASSED")