Time Series Forecasting LSTM Sequence Models Temporal Dynamics
# Time Series Forecasting: LSTM Sequence Models & Temporal Dynamics
## Introduction & Motivation
LSTM/GRU networks capture temporal dependencies in sequences. Encoder-decoder architecture: encode history; decode forecasts. Attention mechanisms highlight relevant past steps. Handles variable-length inputs, nonlinear dynamics. Critical for stock prediction, weather, demand forecasting.
Motivation: Classical ARIMA assumes linearity; insufficient for complex temporal patterns. RNNs learn nonlinear dynamics; LSTMs avoid vanishing gradients.
Applications: Financial forecasting, weather prediction, traffic flow, electricity demand.
---
## Core Concepts & Theory
### LSTM Architecture
Forget gate: decide what to discard. Input gate: what to add. Output gate: what to output. Cell state propagates information across time.
### Seq2Seq with Attention
Encoder summarizes history; decoder generates forecast. Attention mechanism learns to focus on relevant past steps.
### Multi-Horizon Forecasting
Single-step vs. multi-step ahead. Autoregressive (feed prediction back) vs. direct (predict all steps).
---
## Mathematical Formulation
LSTM forward pass:
$$f_t = \sigma(W_f h_{t-1} + U_f x_t + b_f)$$
$$i_t = \sigma(W_i h_{t-1} + U_i x_t + b_i)$$
$$ ilde{C}_t = anh(W_c h_{t-1} + U_c x_t + b_c)$$
$$C_t = f_t \odot C_{t-1} + i_t \odot ilde{C}_t$$
$$o_t = \sigma(W_o h_{t-1} + U_o x_t + b_o)$$
$$h_t = o_t \odot anh(C_t)$$
---
## Advanced Theory & Extensions
### Temporal Convolutional Networks (TCN)
Dilated convolutions for large receptive fields; parallel training.
### Transformer for Time Series
Self-attention captures long-range dependencies; no recurrence.
### Hierarchical Time Series Forecasting
Reconcile forecasts across hierarchy (summing constraints).
---
## Computational Considerations
LSTM: O(T × d × h) where T = sequence length, d = input dim, h = hidden dim.
Seq2Seq: O(T_in + T_out) × (encoder + decoder complexity).
Training: Gradient clipping prevents exploding gradients; stateful/stateless options.
---
## Practical Implementation Strategies
### Normalization
Standardize time series: μ=0, σ=1. Per-variable or global.
### Context Window
Choose lookback length (e.g., 12 months for annual patterns); tradeoff: longer = more signal but more compute.
### Residual Connections
Skip connections improve deep networks (50+ layers).
---
## Benchmark Datasets & Evaluation
UCI Time Series: 85 datasets; electricity, traffic.
M4 Competition: 100K series; various granularities.
Metrics: MAE, RMSE, MAPE, sMAPE; hierarchical metrics for reconciliation.
---
## Key Challenges & Limitations
### Non-Stationarity
Distributions shift over time; adaptation required.
### Trend & Seasonality
Multiple frequencies; decomposition helps.
### Cold Start
Few historical points; meta-learning or transfer learning.
---
## Hyperparameter Tuning
LSTM layers: 1-3.
Hidden size: 32-256.
Dropout: 0.2-0.5.
Learning rate: 0.001-0.01.
---
## Real-World Applications & Case Studies
Uber: Demand forecasting; spatio-temporal LSTM.
Netflix: Content recommendation; implicit temporal dynamics.
Power Grid: Load forecasting; hierarchical reconciliation.
---
## Integration with Other Methods
Forecasting + Uncertainty → Bayesian LSTM, quantile regression.
Forecasting + Transfer → pretrain on large dataset; finetune.
---
## Summary & Key Takeaways
LSTM/Seq2Seq networks capture temporal dependencies for accurate forecasting via memory cells, attention, and hierarchical reconciliation.
Principles:
1. LSTM cells maintain memory; forget/input/output gates control flow.
2. Seq2Seq encoder-decoder for variable-length inputs/outputs.
3. Attention highlights relevant history.
4. Normalization critical; residual connections stabilize deep networks.
5. Multi-horizon forecasting requires careful loss design.
---
---
## Appendix: Practical Labs
### Lab 1: Basic LSTM Forecasting
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
class LSTMForecaster(nn.Module):
def __init__(self, input_dim=1, hidden_dim=32, output_dim=1):
super().__init__()
self.lstm = nn.LSTM(input_dim, hidden_dim, batch_first=True)
self.linear = nn.Linear(hidden_dim, output_dim)
def forward(self, x):
lstm_out, _ = self.lstm(x)
forecast = self.linear(lstm_out[:, -1, :])
return forecast
# Data
np.random.seed(42)
t = np.arange(100)
y = np.sin(t * 0.1) + 0.1 * np.random.randn(100)
y = (y - y.mean()) / (y.std() + 1e-8)
# Sliding window
window_size = 12
X, Y = [], []
for i in range(len(y) - window_size):
X.append(y[i:i+window_size])
Y.append(y[i+window_size])
X = torch.FloatTensor(X).unsqueeze(-1)
Y = torch.FloatTensor(Y).unsqueeze(-1)
model = LSTMForecaster(input_dim=1, hidden_dim=32, output_dim=1)
optimizer = optim.Adam(model.parameters(), lr=0.01)
criterion = nn.MSELoss()
losses = []
for epoch in range(30):
output = model(X)
loss = criterion(output, Y)
optimizer.zero_grad()
loss.backward()
optimizer.step()
losses.append(loss.item())
final_mse = losses[-1]
print(f"Final MSE: {final_mse:.4f}")
assert final_mse < 1.0, "MSE should be reasonable"
assert len(losses) == 30, "Should have 30 epochs"
print("✓ LSTM forecasting working")
if __name__ == "__main__":
print("Lab 1: LSTM Forecast - PASSED")### Lab 2: Multi-Step Ahead Forecasting
import torch
import torch.nn as nn
import numpy as np
class MultiStepLSTM(nn.Module):
def __init__(self, input_dim=1, hidden_dim=32, steps_ahead=5):
super().__init__()
self.lstm = nn.LSTM(input_dim, hidden_dim, batch_first=True)
self.linear = nn.Linear(hidden_dim, steps_ahead)
def forward(self, x):
lstm_out, _ = self.lstm(x)
forecasts = self.linear(lstm_out[:, -1, :])
return forecasts
np.random.seed(42)
t = np.arange(100)
y = np.sin(t * 0.1) + 0.05 * np.random.randn(100)
y = (y - y.mean()) / (y.std() + 1e-8)
X, Y = [], []
window_size, steps = 12, 5
for i in range(len(y) - window_size - steps):
X.append(y[i:i+window_size])
Y.append(y[i+window_size:i+window_size+steps])
X = torch.FloatTensor(X).unsqueeze(-1)
Y = torch.FloatTensor(Y)
model = MultiStepLSTM(input_dim=1, hidden_dim=32, steps_ahead=5)
output = model(X)
print(f"Multi-step output shape: {output.shape}")
assert output.shape == (len(X), 5), "Should forecast 5 steps ahead"
assert torch.isfinite(output).all(), "Output should be finite"
print("✓ Multi-step forecasting working")
if __name__ == "__main__":
print("Lab 2: Multi-Step - PASSED")### Lab 3: Sequence-to-Sequence Forecasting
import torch
import torch.nn as nn
import numpy as np
class Seq2SeqLSTM(nn.Module):
def __init__(self, input_dim=1, hidden_dim=32, output_steps=5):
super().__init__()
self.encoder = nn.LSTM(input_dim, hidden_dim, batch_first=True)
self.decoder = nn.LSTM(1, hidden_dim, batch_first=True)
self.linear = nn.Linear(hidden_dim, 1)
self.output_steps = output_steps
def forward(self, x):
_, (h, c) = self.encoder(x)
decoder_input = torch.zeros(x.size(0), 1, 1)
forecasts = []
for _ in range(self.output_steps):
out, (h, c) = self.decoder(decoder_input, (h, c))
forecast = self.linear(out)
forecasts.append(forecast)
decoder_input = forecast
return torch.cat(forecasts, dim=1)
np.random.seed(42)
y = np.cumsum(np.random.randn(100)) / 10
y = (y - y.mean()) / (y.std() + 1e-8)
X, Y = [], []
window_size, output_steps = 10, 5
for i in range(len(y) - window_size - output_steps):
X.append(y[i:i+window_size])
Y.append(y[i+window_size:i+window_size+output_steps])
X = torch.FloatTensor(X).unsqueeze(-1)
Y = torch.FloatTensor(Y).unsqueeze(-1)
model = Seq2SeqLSTM(input_dim=1, hidden_dim=32, output_steps=5)
output = model(X)
print(f"Seq2Seq output shape: {output.shape}")
assert output.shape == (len(X), 5, 1), "Should match batch and steps"
assert torch.isfinite(output).all(), "Output should be finite"
print("✓ Seq2Seq forecasting working")
if __name__ == "__main__":
print("Lab 3: Seq2Seq - PASSED")### Lab 4: Forecasting Evaluation Metrics
import numpy as np
def mae(y_true, y_pred):
return np.mean(np.abs(y_true - y_pred))
def rmse(y_true, y_pred):
return np.sqrt(np.mean((y_true - y_pred)**2))
def mape(y_true, y_pred):
return np.mean(np.abs((y_true - y_pred) / (y_true + 1e-8))) * 100
# Data
np.random.seed(42)
y_true = np.sin(np.arange(50) * 0.1) + np.random.randn(50) * 0.1
y_pred = y_true + np.random.randn(50) * 0.2
mae_val = mae(y_true, y_pred)
rmse_val = rmse(y_true, y_pred)
mape_val = mape(y_true, y_pred)
print(f"MAE: {mae_val:.4f}, RMSE: {rmse_val:.4f}, MAPE: {mape_val:.2f}%")
assert mae_val > 0, "MAE should be positive"
assert rmse_val >= mae_val, "RMSE >= MAE"
assert mape_val > 0, "MAPE should be positive"
print("✓ Forecasting metrics working")
if __name__ == "__main__":
print("Lab 4: Metrics - PASSED")