Time Series Forecasting Arima Exponential Smoothing Prophet and Deep Learning
# Time Series Forecasting: ARIMA, Exponential Smoothing, Prophet, and Deep Learning
## 1. Introduction & Motivation
Time series forecasting predicts future values based on historical observations. Applications span numerous domains:
- Finance: Stock prices, exchange rates
- Energy: Electricity demand forecasting
- Retail: Sales and inventory prediction
- Weather: Temperature and precipitation
- Healthcare: Patient admission rates
Time series presents unique challenges: temporal dependencies, seasonality, trends, and non-stationarity. Unlike i.i.d. data, observations are correlated with recent history.
This article comprehensively covers classical methods (ARIMA, exponential smoothing), modern approaches (Prophet, deep learning), and practical implementation considerations.
## 2. Core Concepts & Theory
### 2.1 Time Series Components
Additive decomposition:
$$y_t = ext{Trend}_t + ext{Seasonal}_t + ext{Residual}_t$$
- Trend: Long-term direction
- Seasonal: Regular periodic patterns (daily, weekly, yearly)
- Residual: Irregular noise
Multiplicative:
$$ y_t = ext{Trend}_t imes ext{Seasonal}_t imes ext{Residual}_t $$
### 2.2 Stationarity
Stationary process has constant mean, variance, autocorrelation over time.
Non-stationary: Mean/variance change, trends exist.
ARIMA requires stationarity; differencing converts non-stationary to stationary:
$$\Delta y_t = y_t - y_{t-1}$$
Higher-order differencing:
$$ \Delta^2 y_t = \Delta y_t - \Delta y_{t-1} $$
### 2.3 ARIMA Models
AutoRegressive Integrated Moving Average combines:
- AR(p): Autoregressive, uses past values
- I(d): Integrated, differencing order
- MA(q): Moving average, uses past errors
$$y_t = \phi_1 y_{t-1} + \cdots + \phi_p y_{t-p} + \epsilon_t + heta_1 \epsilon_{t-1} + \cdots + heta_q \epsilon_{t-q}$$
ARIMA(p,d,q) combines all three components.
### 2.4 Exponential Smoothing
Simple Exponential Smoothing (SES):
$$\hat{y}_{t+1} = \alpha y_t + (1-\alpha) \hat{y}_t$$
where
$$ \alpha \in [0,1] $$
is smoothing coefficient. High
$$ \alpha $$
: recent values weighted more.
Generalization: Holt-Winters includes trend and seasonality.
## 3. Mathematical Formulation
### 3.1 ARIMA Equations
AR(p) component:
$$y_t = c + \sum_{i=1}^{p} \phi_i y_{t-i} + \epsilon_t$$
MA(q) component:
$$y_t = \mu + \sum_{i=1}^{q} heta_i \epsilon_{t-i} + \epsilon_t$$
Combined ARIMA(p,d,q):
$$\Delta^d y_t = c + \sum_{i=1}^{p} \phi_i \Delta^d y_{t-i} + \sum_{i=1}^{q} heta_i \epsilon_{t-i} + \epsilon_t$$
### 3.2 Exponential Smoothing with Seasonality
Holt-Winters multiplicative:
$$\hat{y}_{t+h} = (\ell_t + h b_t) s_{t+h-m}$$
where:
-
$$ \ell_t $$
: Level (trend)
-
$$ b_t $$
: Slope
-
$$ s_t $$
: Seasonal component
- m: Seasonal period (12 for monthly, 365 for daily)
Update equations:
$$\ell_t = \alpha \frac{y_t}{s_{t-m}} + (1-\alpha)(\ell_{t-1} + b_{t-1})$$
$$b_t = \beta(\ell_t - \ell_{t-1}) + (1-\beta)b_{t-1}$$
$$s_t = \gamma \frac{y_t}{\ell_t} + (1-\gamma) s_{t-m}$$
### 3.3 ACF and PACF for Model Selection
Autocorrelation Function (ACF): Correlation between observations at different lags:
$$ ho_k = \frac{\sum_{t=1}^{n-k}(y_t - \bar{y})(y_{t+k} - \bar{y})}{\sum_{t=1}^{n}(y_t - \bar{y})^2}$$
Partial ACF (PACF): Correlation controlling for intermediate lags.
ACF/PACF patterns guide ARIMA order selection:
- ACF cuts off at lag q → MA(q)
- PACF cuts off at lag p → AR(p)
### 3.4 LSTM for Time Series
LSTM cell processes sequence, hidden state captures temporal dependencies:
$$h_t = ext{LSTM}(y_t, h_{t-1})$$
One-step-ahead prediction:
$$\hat{y}_{t+1} = W_o h_t + b_o$$
Multi-step-ahead: Recursive (teacher forcing during training, autoregressive at test).
## 4. Advanced Theory & Extensions
### 4.1 SARIMA (Seasonal ARIMA)
Includes seasonal differencing and seasonal AR/MA terms:
$$ ext{SARIMA}(p,d,q)(P,D,Q)_m$$
Example:
$$ (1,1,1)(1,1,1)_{12} $$
for monthly data with annual seasonality.
Seasonal differencing:
$$ \Delta_m y_t = y_t - y_{t-m} $$
### 4.2 Vector Autoregression (VAR)
Multivariate extension, models multiple time series jointly:
$$y_t = c + \sum_{i=1}^{p} A_i y_{t-i} + \epsilon_t$$
where
$$ y_t \in \mathbb{R}^k $$
(k series),
$$ A_i $$
are coefficient matrices.
Benefits:
- Captures cross-series dependencies
- Can improve forecasting through mutual information
- Useful for related series (e.g., multiple weather stations)
### 4.3 Transformer Models for Time Series
Attention-based architectures designed for time series:
$$ ext{Attention}(Q, K, V) = ext{softmax}\left(\frac{QK^T}{\sqrt{d_k}} ight)V$$
Applied to time series:
- Queries: Future position
- Keys/Values: Historical observations
- Learns which past observations matter for future
Benefits over LSTM:
- Parallel computation (LSTM sequential)
- Better long-range dependencies
- Multi-head attention for multiple patterns
### 4.4 Probabilistic Forecasting
Rather than point estimate, predict full distribution:
$$p(y_{t+h} | y_{1:t})$$
Methods:
- Quantile regression: Predict specific percentiles (0.1, 0.5, 0.9)
- Gaussian processes: Output distribution with uncertainty
- Deep probabilistic: Neural network outputs mean + variance
Enables uncertainty quantification, crucial for risk management.
## 5. Computational Considerations
### 5.1 Time Complexity
ARIMA:
- Fitting:
$$ O(n \cdot p \cdot q) $$
where n is series length
- Typical: <1s for
$$ n=1000 $$
,
$$ p,q \leq 10 $$
Exponential Smoothing:
- Fitting: O(n) very fast
- Typical: <10ms
LSTM:
- Fitting:
$$ O(n \cdot h^2) $$
where h is hidden size
- 1 epoch on 1000 timesteps,
$$ h=128 $$
: ~100ms
Prophet:
- Fitting: O(n) for trend, seasonal components
- Typical: ~1s for 1000 observations
### 5.2 Memory Requirements
ARIMA:
$$ O(p + q) $$
modest
LSTM:
$$ O(n \cdot h) $$
for sequence storage during training
Transformer:
$$ O(n^2) $$
for attention matrix (scales poorly for long sequences)
### 5.3 Inference Speed
ARIMA: <1ms per forecast (very fast)
Exponential smoothing: <1ms per forecast
LSTM: ~10-50ms per forecast (depends on sequence length)
Transformer: ~50-200ms per forecast
ARIMA/ES ideal for real-time forecasting; deep learning slower.
## 6. Practical Implementation Strategies
### 6.1 Data Preprocessing
Normalization:
$$ ilde{y}_t = \frac{y_t - \mu}{\sigma}$$
Important for LSTM/Transformer (gradient-based); less critical for ARIMA.
Handling missing values:
- Forward fill:
$$ y_t = y_{t-1} $$
if missing
- Interpolation: Linear, spline
- Model imputation: Use ARIMA/exponential smoothing to estimate
Outlier handling:
- Winsorization: Cap extreme values
- Robust scaling: Use median/IQR instead of mean/std
- Detection + modeling: Explicitly model outliers
### 6.2 Train-Test Split
Unlike i.i.d. data, time series requires careful splitting:
Correct (temporal order):
Train: [1:500], Test: [501:600]Incorrect (random shuffle):
Train: [random 70%], Test: [random 30%] # Leaks future infoFor multiple-step-ahead evaluation:
Train: [1:400], Validation: [401:500], Test: [501:600]### 6.3 ARIMA Order Selection
AIC/BIC: Information criteria balancing fit and complexity
$$ ext{AIC} = -2 \log L + 2k$$
where k is number of parameters.
Grid search: Try combinations of (p,d,q) and select best AIC
Best ARIMA: Grid over p,d,q in [0,5]Auto ARIMA: Automated selection (e.g., `pmdarima` library)
### 6.4 Seasonality Detection
Seasonal strength: How much of variance explained by seasonality?
$$ ext{SeasonalStrength} = 1 - \frac{ ext{Var}( ext{Residual})}{ ext{Var}( ext{Detrended})}$$
If strong seasonality: Use SARIMA/Holt-Winters
If weak: Standard ARIMA sufficient
## 7. Benchmark Datasets & Evaluation
### 7.1 Forecasting Benchmarks
M4 Competition:
- 100K time series of varying frequencies
- ARIMA, ES, and hybrids competitive
- Deep learning methods show marginal improvements
Electricity Load:
- 370 series, 15-min resolution, 4 years
- Strong daily/weekly seasonality
- Baseline: Persistence model (forecast = last value)
- ARIMA: MAPE 3-4%, MAE 0.3-0.4
Tourism Dataset:
- 366 series, monthly, 48 years
- Varying trend/seasonality
- SARIMA: MAPE 10-15%
- Deep learning: 8-12% MAPE
### 7.2 Evaluation Metrics
MAPE (Mean Absolute Percentage Error):
$$ ext{MAPE} = \frac{1}{n} \sum_{t=1}^{n} \frac{|y_t - \hat{y}_t|}{|y_t|}$$
Scale-invariant, interpretable as percentage error. Issues with small values (division by small number).
MAE (Mean Absolute Error):
$$ ext{MAE} = \frac{1}{n} \sum_{t=1}^{n} |y_t - \hat{y}_t|$$
Scale-dependent but stable.
RMSE (Root Mean Square Error):
$$ ext{RMSE} = \sqrt{\frac{1}{n} \sum_{t=1}^{n} (y_t - \hat{y}_t)^2}$$
Emphasizes large errors.
### 7.3 Benchmark Results
Electricity Load (L=24 hour ahead):
- Naive: MAPE 4.2%
- ARIMA: MAPE 3.5%
- Exponential smoothing: MAPE 3.4%
- LSTM: MAPE 3.2% (modest improvement)
- Prophet: MAPE 3.6%
## 8. Key Challenges & Limitations
### 8.1 Non-Stationarity
Real-world time series often non-stationary (trends, level shifts). ARIMA requires differencing, which:
- Works when trend linear
- Fails with structural breaks (regime changes)
- Requires manual order selection
### 8.2 Seasonal Complexity
Simple seasonality (fixed pattern repeating) handled well. Challenges:
- Multiple seasonalities (hourly + daily + yearly)
- Changing seasonality (amplitude varies over time)
- Holiday effects (not regular)
Prophet handles these better; ARIMA struggles.
### 8.3 Distribution Shifts
Models trained on historical data may not generalize to:
- Trend changes (e.g., COVID-19 impact on mobility)
- New seasonal patterns
- Sudden level shifts
Retraining helps but requires labeled recent data.
### 8.4 Sparse/Irregular Data
Many real time series:
- Missing values
- Irregular spacing (trades in finance)
- Bursts (social media activity)
Standard methods assume regular spacing.
## 9. Hyperparameter Tuning & Optimization
### 9.1 ARIMA Hyperparameters
Differencing order (d): 0, 1, or 2 typically
- d=0: Already stationary
- d=1: Single differencing
- d=2: Rare, usually overfitting
AR order (p): Usually 0-5
- Test using PACF
- AIC selection automatic
MA order (q): Usually 0-5
- Test using ACF
- Larger q = more error terms
### 9.2 Exponential Smoothing Parameters
Alpha (level smoothing): 0.01-1
- Higher: Recent values weighted more
- Typical: 0.1-0.3
Beta (trend smoothing): 0.01-1 (if using trend)
- Lower: Smoother trend
- Typical: 0.01-0.1
Gamma (seasonal smoothing): 0.01-1 (if using seasonality)
- Typical: 0.01-0.1
### 9.3 Prophet Hyperparameters
Changepoint prior scale: 0.001-0.5 (flexibility of trend)
- Higher: More trend changes
- Default: 0.05
Seasonality strength: 0-1 (weight on seasonality)
- Higher: Stronger seasonality
- Default: 10
Interval width: Width of confidence intervals (0-1)
- Default: 0.8 (80% interval)
### 9.4 LSTM Hyperparameters
Sequence length: 24-168 typical (hours, days)
- Longer: More context but more parameters
- Typical: 24-48 for hourly data
Hidden size: 32-256 typical
- Larger: More capacity, risk of overfitting
- Typical: 64-128
Dropout: 0.2-0.5 on hidden states
- Prevents overfitting
- Typical: 0.3
Learning rate: 0.001-0.01
- Lower: Safer but slower convergence
- Typical: 0.001
## 10. Real-World Applications & Case Studies
### 10.1 Electricity Demand Forecasting
Problem: Forecast hourly electricity demand for 10M+ homes
Setup:
- Data: 3-year hourly demand, 15-min resolution
- Seasonality: Daily, weekly, yearly (strong)
- Trend: Growing but with temporary disruptions
- Requirement: MAPE < 3%
Method Comparison:
- ARIMA(1,0,1)(1,1,1)_24: MAPE 3.8%
- Exponential smoothing: MAPE 3.5%
- Prophet: MAPE 3.3%
- LSTM (h=128, L=48): MAPE 3.1%
Deployment:
- Prophet selected: Good balance of accuracy and interpretability
- Trend changepoints: Automatic detection of regime changes
- Hourly retraining: Update model with new observations
Key insights:
- Ensemble (Prophet + LSTM): MAPE 3.0%
- Holiday effects critical (must model explicitly)
- Temperature correlation: Include as external regressor
### 10.2 Stock Price Forecasting
Problem: Predict daily stock price changes
Setup:
- Data: 5 years daily OHLC (open, high, low, close)
- No clear seasonality (random walk)
- High noise, difficult to forecast
Method Comparison:
- ARIMA: MAPE 5-7% (captures some structure)
- Random walk baseline: MAPE 4-5%
- LSTM: MAPE 4-6% (slight improvement, inconsistent)
Deployment:
- Trading strategy: Difficult to beat buy-and-hold
- LSTM used for ensemble, not standalone
- Focus on risk prediction (volatility forecasting) instead
Lessons:
- Time series forecasting harder than reputation suggests
- Markets are noisy; traditional methods competitive
- Deep learning helps for noisy/complex patterns but not guaranteed
### 10.3 Website Traffic Forecasting
Problem: Forecast hourly website traffic for capacity planning
Setup:
- Data: 2 years hourly traffic, 24K sites
- Seasonality: Very strong (daily patterns)
- Trend: Variable (depends on business model)
- Requirement: MAPE < 10%
Architecture:
- Global model: Forecast across all sites
- Site-specific model: Fine-tune for individual sites
- Hybrid: Blend global + site-specific
Results:
- Global ARIMA: MAPE 8-12% (too generic)
- Site-specific Prophet: MAPE 6-8%
- Site-specific LSTM: MAPE 5-7%
- Ensemble: MAPE 5%
Deployment:
- Automated retraining weekly
- Trend changepoint detection for promotions
- Anomaly detection (traffic spikes)
### 10.4 COVID-19 Case Forecasting
Problem: Forecast daily COVID cases during pandemic
Setup:
- Data: 500+ day series, weekly to monthly trends
- Non-stationary: Multiple waves with different slopes
- Structural breaks: Lockdowns change dynamics
- External regressors: Mobility, vaccination rates
Challenges:
- Distribution shift: Past patterns become invalid
- Concept drift: Population behavior changes
- Long-term forecasting difficult (>30 days)
Method:
- ARIMA failed (non-stationary, structural breaks)
- Prophet: Handles changepoints well, MAPE ~15% (7 days)
- Exponential smoothing: ~10% MAPE (7 days)
- LSTM (with external features): ~9% MAPE (7 days)
Key insight:
- Ensemble of methods crucial during rapid change
- Retraining on recent data (last 30-60 days) critical
- Uncertainty intervals matter (not just point forecasts)
## 11. Integration with Other Methods
### 11.1 Ensemble Forecasting
Combine multiple models:
$$\hat{y}_{t+h} = w_1 \hat{y}^{ ext{ARIMA}} + w_2 \hat{y}^{ ext{ES}} + w_3 \hat{y}^{ ext{LSTM}}$$
Typical improvement: 5-15% over best single model.
### 11.2 External Regressors
Include causally-related features:
$$y_t = f(y_{t-1}, \ldots, x_{1,t}, x_{2,t}, \ldots)$$
Example: Temperature in demand forecasting
- Improves MAPE by 10-20%
- Requires forecasting regressors too (or use realized)
### 11.3 Anomaly Detection
Detect and handle outliers:
$$\hat{\epsilon}_t = y_t - \hat{y}_t$$
If
$$ |\hat{\epsilon}_t| > 3\sigma $$
: Flag as anomaly
- Remove for training
- Add explicitly to model
### 11.4 Meta-Learning for Forecasting
Learn model selection across many time series:
- Feature extraction: Trend strength, seasonality, autocorrelation
- Meta-model: Predict best method for new series
- Result: Automated method selection
## 12. Future Research Directions
### 12.1 Transformer Models at Scale
Current: Limited to <10K length sequences
Goal: Efficient attention for very long sequences
### 12.2 Uncertainty Quantification
Move beyond point forecasts to full distributions:
- Quantile regression
- Gaussian processes
- Bayesian deep learning
### 12.3 Transfer Learning for Time Series
Leverage patterns from one domain to another:
- Energy to traffic forecasting (both have strong daily patterns)
- Cross-site transfer (website traffic)
- Pre-training on large datasets
### 12.4 Online Learning
Continuous adaptation to new data:
- Streaming models
- Concept drift handling
- Minimal retraining overhead
## 13. Summary & Key Takeaways
Classical Methods:
- ARIMA: Interpretable, ~3-5% MAPE (electricity), requires stationarity
- Exponential Smoothing: Fast (<1ms), ~3-4% MAPE, good for seasonal data
- Prophet: Automatic seasonality/trends, ~3-4% MAPE, interpretable components
Deep Learning Methods:
- LSTM: Good for complex patterns, ~2-3% MAPE, but slower to train/infer
- Transformer: Good attention mechanism, emerging for time series
- Ensemble: Combining methods gives 5-15% improvement
Performance:
- Simple series (electricity): Classical methods competitive (~3% MAPE)
- Complex series (stocks): Deep learning modest advantage (<1%)
- Real deployment: Often use ensemble
Hyperparameters:
- ARIMA: Grid search (p,d,q) based on AIC
- ES: Alpha 0.1-0.3, beta 0.01-0.1
- Prophet: Changepoint prior 0.05, seasonality strength 10
- LSTM: Sequence 24-48, hidden 64-128, dropout 0.3
Practical Considerations:
- Preprocessing: Normalization for neural networks, less critical for classical
- Seasonality: Must handle explicitly (SARIMA, Holt-Winters, Prophet)
- External features: 10-20% improvement if correlated
- Retraining: Weekly to monthly depending on dynamics
- Ensemble: Always try combining different methods
When to Use:
- ARIMA: Interpretability required, stationary data
- Prophet: Holiday effects, multiple seasonalities
- LSTM: Complex, non-stationary patterns
- Exponential smoothing: Real-time constraints
Most practical deployments use Prophet or exponential smoothing for balance of accuracy, speed, and interpretability. Deep learning increasingly popular for complex patterns but requires careful tuning.
---
## Appendix: Practical Implementation Labs
### Lab 1: ARIMA Forecasting
from statsmodels.tsa.arima.model import ARIMA
import numpy as np
def fit_arima_forecast(y, p=1, d=1, q=1, forecast_steps=12):
"""Fit ARIMA and forecast"""
model = ARIMA(y, order=(p, d, q))
fitted_model = model.fit()
# Forecast
forecast = fitted_model.get_forecast(steps=forecast_steps)
forecast_values = forecast.predicted_mean
confidence_int = forecast.conf_int(alpha=0.05)
return forecast_values, confidence_int, fitted_model
def auto_arima_select(y):
"""Auto select best ARIMA order"""
from pmdarima import auto_arima
model = auto_arima(y, seasonal=False, stepwise=True)
return model.order### Lab 2: Exponential Smoothing
from statsmodels.tsa.holtwinters import ExponentialSmoothing
def fit_exponential_smoothing(y, trend='add', seasonal=None, seasonal_periods=12):
"""Fit exponential smoothing model"""
model = ExponentialSmoothing(y, trend=trend, seasonal=seasonal,
seasonal_periods=seasonal_periods)
fitted_model = model.fit()
# Forecast
forecast = fitted_model.forecast(steps=12)
return forecast, fitted_model### Lab 3: Prophet Forecasting
from prophet import Prophet
import pandas as pd
def forecast_with_prophet(df, periods=30):
"""Forecast with Prophet"""
# df must have columns: ds (date), y (value)
model = Prophet(changepoint_prior_scale=0.05,
seasonality_mode='additive')
model.fit(df)
future = model.make_future_dataframe(periods=periods)
forecast = model.predict(future)
return forecast### Lab 4: LSTM Time Series Forecasting
import torch
import torch.nn as nn
class LSTMForecaster(nn.Module):
def __init__(self, input_size, hidden_size, num_layers=1, output_size=1):
super().__init__()
self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
self.fc = nn.Linear(hidden_size, output_size)
def forward(self, x):
lstm_out, _ = self.lstm(x)
last_out = lstm_out[:, -1, :]
forecast = self.fc(last_out)
return forecast
def prepare_time_series_data(y, seq_length=24):
"""Prepare data for LSTM"""
X, Y = [], []
for i in range(len(y) - seq_length):
X.append(y[i:i+seq_length])
Y.append(y[i+seq_length])
return torch.tensor(X, dtype=torch.float32), torch.tensor(Y, dtype=torch.float32)
def train_lstm_forecaster(y, seq_length=24, epochs=100):
"""Train LSTM forecaster"""
X, Y = prepare_time_series_data(y, seq_length)
model = LSTMForecaster(input_size=1, hidden_size=64)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.MSELoss()
for epoch in range(epochs):
optimizer.zero_grad()
output = model(X.unsqueeze(-1))
loss = criterion(output.squeeze(), Y)
loss.backward()
optimizer.step()
return model