LSTM and Recurrent Neural Networks Capturing Sequential Dependencies
# LSTM and Recurrent Neural Networks: Capturing Sequential Dependencies
## 1. Introduction & Motivation
Recurrent Neural Networks (RNNs) are fundamental architectures for processing sequential data where the output depends on previous inputs and hidden states. Unlike feedforward networks that process inputs independently, RNNs maintain hidden state vectors that evolve as they process sequences, enabling them to capture temporal dependencies and long-range patterns in data.
Traditional RNNs suffer from critical limitations when dealing with long-term dependencies. During backpropagation through time (BPTT), gradients either explode or vanish as they propagate backward through many time steps, making it difficult to learn dependencies spanning hundreds of steps. Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs) address these issues through sophisticated gating mechanisms that regulate information flow, allowing networks to selectively retain or discard information across time steps.
This article provides comprehensive coverage of recurrent architectures, focusing on LSTM and GRU designs, the mathematical foundations of gradient flow, bidirectional variants, and practical considerations for sequence modeling tasks.
## 2. Core Concepts & Theory
### 2.1 Basic RNN Architecture
A standard RNN processes sequences by maintaining a hidden state h_t that gets updated at each time step:
$$h_t = anh(W_{hx} x_t + W_{hh} h_{t-1} + b_h)$$
$$y_t = W_{yh} h_t + b_y$$
where x_t is the input at time t, h_t is the hidden state, W_hx, W_hh, W_yh are weight matrices, and b_h, b_y are biases.
### 2.2 The Vanishing Gradient Problem
During BPTT, gradients flow backward through time steps. For a loss function L at time T:
$$\frac{\partial L}{\partial h_t} = \frac{\partial L}{\partial h_T} \cdot \frac{\partial h_T}{\partial h_t}$$
The gradient product involves terms like:
$$\frac{\partial h_{t+1}}{\partial h_t} = W_{hh}^T ext{diag}(1 - h_{t+1}^2)$$
When the eigenvalues of W_hh^T are less than 1, repeated multiplication causes gradients to decay exponentially, making early time steps difficult to learn from.
### 2.3 LSTM Architecture
LSTMs overcome vanishing gradients through memory cells and gating mechanisms. The LSTM cell maintains:
- A cell state c_t (long-term memory)
- A hidden state h_t (short-term output)
- Three gates: input, forget, and output
Gates are computed as:
$$i_t = \sigma(W_{xi} x_t + W_{hi} h_{t-1} + b_i)$$
$$f_t = \sigma(W_{xf} x_t + W_{hf} h_{t-1} + b_f)$$
$$o_t = \sigma(W_{xo} x_t + W_{ho} h_{t-1} + b_o)$$
where sigma is the sigmoid function. A candidate memory cell is computed:
$$ ilde{c}_t = anh(W_{xc} x_t + W_{hc} h_{t-1} + b_c)$$
The cell state and hidden state update:
$$c_t = f_t \odot c_{t-1} + i_t \odot ilde{c}_t$$
$$h_t = o_t \odot anh(c_t)$$
### 2.4 GRU Architecture
GRUs simplify LSTMs by combining cell state and hidden state into a single state vector:
$$r_t = \sigma(W_{xr} x_t + W_{hr} h_{t-1} + b_r)$$
$$z_t = \sigma(W_{xz} x_t + W_{hz} h_{t-1} + b_z)$$
$$ ilde{h}_t = anh(W_{xh} x_t + W_{hh} (r_t \odot h_{t-1}) + b_h)$$
$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot ilde{h}_t$$
GRUs have fewer parameters than LSTMs while often achieving comparable performance, making them computationally more efficient.
### 2.5 Bidirectional RNNs
Bidirectional RNNs process sequences in both forward and backward directions, combining hidden states:
$$h_t = ext{concat}(\overrightarrow{h}_t, \overleftarrow{h}_t)$$
This approach is particularly effective for tasks where future context is available (e.g., sequence labeling, machine translation encoding).
## 3. Mathematical Formulation
### 3.1 Backpropagation Through Time (BPTT)
For an LSTM, the gradient with respect to cell state enjoys a crucial advantage. The cell state update c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t creates an additive path that mitigates vanishing gradients:
$$\frac{\partial c_t}{\partial c_{t-1}} = f_t$$
Since forget gates are bounded between 0 and 1, gradients don't explode multiplicatively over long sequences. The gradient flow:
$$\frac{\partial L}{\partial c_t} = \frac{\partial L}{\partial c_{t+1}} \cdot f_{t+1} + \frac{\partial L}{\partial h_t} \cdot o_t \cdot (1 - anh^2(c_t))$$
### 3.2 Gradient Clipping
Despite LSTM improvements, gradient explosion remains possible. Gradient clipping normalizes gradients during training:
$$ ilde{g} = g \quad ext{if } ||g|| \leq heta, \quad ext{else} \quad ilde{g} = \frac{ heta}{||g||} g$$
where theta is a clipping threshold (typically 1 to 5) and ||g|| is the norm of the gradient vector.
### 3.3 Peephole Connections
Peephole LSTM variants add connections from cell state to gates:
$$i_t = \sigma(W_{xi} x_t + W_{hi} h_{t-1} + W_{ci} c_{t-1} + b_i)$$
This allows gates to depend directly on cell state magnitudes, improving learning of precise timing patterns.
## 4. Advanced Theory & Extensions
### 4.1 Attention Mechanisms in RNNs
Attention allows RNNs to dynamically focus on relevant parts of sequences. For each time step t, attention scores are computed:
$$\alpha_{t,s} = \frac{\exp(e_{t,s})}{\sum_{s'} \exp(e_{t,s'})}$$
where e_{t,s} represents relevance between position t and s. The context vector becomes:
$$c_t = \sum_s \alpha_{t,s} h_s$$
### 4.2 Layer Normalization and RNNs
Layer normalization (LN) on LSTM activations stabilizes training:
$$h_t = o_t \odot anh( ext{LN}(c_t))$$
LN operations have shown to improve convergence speed and final performance compared to batch normalization in recurrent settings.
### 4.3 Deep RNNs
Stacking multiple LSTM layers creates deep recurrent architectures:
$$h_t^{(l)} = ext{LSTM}^{(l)}(h_t^{(l-1)}, h_{t-1}^{(l)})$$
Multiple layers learn hierarchical temporal representations, with lower layers capturing local patterns and upper layers modeling long-range dependencies.
### 4.4 Residual Connections
Residual connections between stacked layers improve gradient flow:
$$h_t^{(l)} = h_t^{(l-1)} + \Delta h_t^{(l)}$$
This modification enables training of networks with 10+ stacked layers without performance degradation.
## 5. Computational Considerations
### 5.1 Memory Requirements
LSTM computation is memory-intensive. For a batch of size B, sequence length T, and hidden dimension d_h, the memory requirements per layer are: Activations O(T * B * d_h), Gradients O(T * B * d_h), and Parameters O(4 d_h^2 + 4 d_x d_h).
Storing activations for all time steps during training requires significant memory, making gradient checkpointing necessary for long sequences.
### 5.2 Gradient Checkpointing
Gradient checkpointing trades computation for memory by recomputing activations during backpropagation:
- Save checkpoints at sqrt(T) intervals
- Recompute forward pass for segments during backward pass
- Memory reduction: O(sqrt(T)) instead of O(T)
### 5.3 Inference Speed Optimization
RNN inference is inherently sequential (cannot parallelize over time). Optimization strategies:
- Quantization: 8-bit weights/activations reduce memory and latency
- Pruning: Remove connections with small weights (~50% reduction)
- Knowledge distillation: Transfer to smaller models
- Batch processing: Process multiple sequences simultaneously
### 5.4 Efficient Gate Implementations
Modern implementations fuse computations:
$$[i_t, f_t, ilde{c}_t, o_t] = ext{split}(W(x_t \oplus h_{t-1}) + b)$$
Single matrix multiplication followed by splitting reduces memory bandwidth and kernel launch overhead.
## 6. Practical Implementation Strategies
### 6.1 Initialization Strategies
Proper initialization is crucial for RNN training:
- Orthogonal initialization: Initialize W_hh as orthogonal matrix, scales to maintain spectral radius near 1
- He initialization: Standard deviation sqrt(2/n_in) for input-to-hidden weights
- Uniform initialization: U[-sqrt(k), sqrt(k)] where k = 1/n_in for stability
Orthogonal initialization of recurrent weights significantly accelerates convergence.
### 6.2 Dropout Strategies
Standard dropout in RNNs can disrupt temporal dependencies. Effective variants:
- Recurrent dropout: Apply same mask across time steps for recurrent connections
- Variational dropout: Same dropout mask for h_{t-1} → h_t connections
- Zoneout: Stochastically maintain previous hidden state instead of zeroing
Variational dropout with p=0.5 on hidden state connections works well in practice.
### 6.3 Learning Rate Scheduling
RNNs are sensitive to learning rates. Recommended strategies:
- Exponential decay: η_t = η_0 · 0.96^(t/epoch)
- Warm restart: Cyclical LR with SGDR for fine-tuning
- Adaptive methods: Adam with β_1=0.9, β_2=0.999, initial η ≈ 0.001
Gradient clipping (norm-based, threshold 1-5) should accompany all strategies.
### 6.4 Sequence Length Management
Handling variable-length sequences:
- Padding: Pad to maximum batch length (wasteful)
- Bucketing: Group sequences by length, process separately
- Truncation: Truncate to fixed length, handle multiple chunks
- Masking: Mask padded positions during attention/output computation
Bucketing improves efficiency by ~30% compared to naive padding.
## 7. Benchmark Datasets & Evaluation
### 7.1 Language Modeling
Penn Treebank: 929K training tokens from Wall Street Journal articles
- Typical LSTM performance: ~78 perplexity (1-layer, 200 units)
- State-of-the-art: ~55 perplexity (ensemble methods)
- Evaluation: Bits-per-character (BPC) or perplexity
Wikitext-103: 103M tokens from Wikipedia
- More realistic corpus, diverse topics
- LSTM baseline: ~25 perplexity
- Current SoTA: ~10 perplexity (transformer-based)
### 7.2 Machine Translation
WMT English-German: ~6.7M parallel sentence pairs
- BLEU score: LSTM achieves ~19, transformers ~28
- Attention mechanism essential for competitive performance
- Evaluation: BLEU, METEOR, TER (test set size ~3,000 sentences)
### 7.3 Sequence Tagging
CoNLL 2003 NER: 14,987 training sentences, 4 entity types
- Bidirectional LSTM-CRF: ~90.94 F1-score
- Simple LSTM baseline: ~88.5 F1-score
- Metrics: Precision, Recall, F1-score on entity-level evaluation
### 7.4 Time Series Prediction
Electricity Consuming Load: 15-minute resolution, 321 sequences
- Metrics: MAE, RMSE, MAPE
- LSTM outperforms ARIMA by 20-30% on this dataset
- Sequence length: 140 time steps for prediction horizon
## 8. Key Challenges & Limitations
### 8.1 Gradient Flow Challenges
Despite gating improvements, challenges remain:
- Exploding gradients in GRUs: Less robust than LSTM to extreme values
- Dying RNN problem: Forget gates learn to stay near 1, ignoring input
- Gradient reversal: Some tasks require careful gradient management
Mitigation: Initialize forget gate bias to 1-2, use gradient clipping, monitor gate values during training.
### 8.2 Long-Term Dependencies Beyond Practical Reach
LSTMs still struggle with dependencies >100-200 steps:
- Information decay occurs even with gating mechanisms
- Attention mechanisms required for very long-range context
- This limitation motivated transformer architectures
### 8.3 Computational Inefficiency
- Sequential nature: Cannot parallelize inference over time
- BPTT cost: O(T · B) computations for sequence length T
- Memory bottleneck: Storing full sequences limits batch size
### 8.4 Instability on Certain Tasks
Some applications show training instability:
- Language modeling with small learning rates causes slow convergence
- NMT exhibits exposure bias and reference bias issues
- Reinforcement learning with RNN policies shows high variance
## 9. Hyperparameter Tuning & Optimization
### 9.1 Architecture Hyperparameters
Hidden dimension: Balance between expressiveness and efficiency
- Small datasets: d_h = 128–256
- Medium datasets: d_h = 512–1024
- Large datasets: d_h = 1024–2048
- Diminishing returns beyond d_h = 2048
Number of layers: Typically 1-3 for standard tasks
- Single layer sufficient for simple sequence labeling
- 2-3 layers needed for language modeling
- Beyond 4 layers: residual connections essential
### 9.2 Regularization Hyperparameters
Dropout rate: Apply to all but last layer
- p = 0.1–0.3 typically optimal
- Varies by task: lower for small datasets, higher for large
- Recurrent dropout (p=0.2–0.4) more aggressive than standard
Weight decay: λ = 1e-5 to 1e-4
- Prevents overfitting on medium datasets
- Not always beneficial for large datasets
- Interact with batch size: smaller batches need less regularization
### 9.3 Optimization Hyperparameters
Batch size: B = 16–128 typical
- Larger batches (64-128) for computational efficiency
- Smaller batches (16-32) for better generalization
- Memory permitting, gradient accumulation simulates larger batches
Learning rate: Task and optimizer dependent
- SGD with momentum: η = 0.01–0.1
- Adam: η = 0.0001–0.001
- RMSprop: η = 0.0001–0.01
Gradient clipping: ||g||_max = 1–5
- Essential for BPTT stability
- Typical choice: clip to norm 1 or 5
- Monitor clipping frequency; should affect <5% of updates
## 10. Real-World Applications & Case Studies
### 10.1 Machine Translation (Neural Machine Translation)
Problem: Translate English text to German with sequence-to-sequence architecture
Architecture:
- Encoder: 2-layer bidirectional LSTM (d_h=512)
- Decoder: 2-layer LSTM with attention
- Attention: Bahdanau additive attention over encoder states
Results:
- Baseline RNN (no attention): 13.2 BLEU
- With attention: 18.5 BLEU (+40% improvement)
- With beam search (size 5): 19.1 BLEU
Lessons learned:
- Attention mechanism critical for alignment
- Bidirectional encoder captures full context
- Large vocabulary (50K tokens) requires subword tokenization
### 10.2 Named Entity Recognition
Problem: Label person, organization, location entities in text
Architecture:
- BiLSTM encoder: 1-layer bidirectional LSTM
- CRF decoder: Handles tag constraints (e.g., no I-tag without B-tag)
- Embeddings: 100-D pretrained word vectors + 30-D character CNN
Results:
- Baseline (BiLSTM only): 89.2 F1
- With CRF: 90.5 F1 (+1.5% improvement)
- With character CNN: 91.2 F1 (+1.9% improvement)
Lessons learned:
- Char-level features crucial for rare words
- CRF constraints enforce meaningful tag sequences
- BiLSTM benefits greatly from pretrained embeddings
### 10.3 Speech Recognition
Problem: Transcribe audio spectrograms to text
Architecture:
- Input: 80-dim MFCC features, frame rate 10ms
- BiLSTM: 3 layers, 512 units each
- CTC loss for alignment-free training
Results:
- Word error rate (WER): 6.7% on test set
- Competitive with GMM-HMM baseline
- End-to-end training much simpler than traditional pipeline
Lessons learned:
- CTC loss handles variable-length alignments
- Multiple stacked layers essential for acoustic modeling
- Bidirectional processing prevents information loss
### 10.4 Time Series Forecasting
Problem: Predict electricity load 24 hours ahead using historical data
Architecture:
- Input: Univariate or multivariate time series
- LSTM: 1-2 layers with dropout (p=0.2)
- Output: Single or multiple time steps ahead
Results:
- MAPE: 2.5% (competitive with Prophet, better than ARIMA)
- RMSE: 0.42 (normalized units)
- Training time: ~5 minutes on CPU
Lessons learned:
- Stationarity preprocessing improves convergence
- Multivariate inputs (temperature, weekday) reduce error
- Dropout essential to prevent overfitting on seasonal patterns
## 11. Integration with Other Methods
### 11.1 Combining with Attention Mechanisms
Attention transforms RNNs into powerful sequence models:
$$ ext{Attention}(Q, K, V) = ext{softmax}\left(\frac{QK^T}{\sqrt{d_k}} ight) V$$
Query from decoder, keys/values from encoder. Multi-head attention (8-12 heads) further improves expressiveness.
### 11.2 Hybrid Architectures with CNNs
CNNs extract local features, RNNs model temporal dependencies:
$$h_t = ext{LSTM}( ext{CNN}(x_t), h_{t-1})$$
Application: Character-level neural language modeling
- CNN layer: Filter sizes 3-7, captures n-gram patterns
- LSTM layer: Captures long-range dependencies
- Performance: 5-10% perplexity improvement over pure LSTM
### 11.3 Multi-Modal Learning
RNNs integrate diverse input modalities:
- Vision-language: CNN for images, LSTM for text
- Audio-visual: Speech RNN + Visual RNN, shared classifier
- Fusion strategies: Early (concatenate inputs), late (combine outputs), hybrid
### 11.4 Reinforcement Learning with RNNs
RNNs as policy networks in RL agents:
$$\pi(a|s) = ext{softmax}(W_{policy} h_t + b_{policy})$$
Where h_t is LSTM hidden state. Useful for partially observable environments and memory requirements.
## 12. Future Research Directions
### 12.1 Improved Gating Mechanisms
Current research explores:
- Coupled input-forget gates: Reduce parameters
- Adaptive gating: Gates with learnable activation functions
- Neural architecture search: Automatically discover optimal gating patterns
### 12.2 Scaling to Extremely Long Sequences
Key challenges:
- Transformer networks address sequential nature but quadratic attention
- State Space Models (S4, Mamba) show O(N) complexity
- Hierarchical RNNs for multi-scale temporal modeling
### 12.3 Interpretability and Visualization
Emerging techniques:
- Gate analysis: Understand what input gates, forget gates, output gates learn
- Attention visualization: Map learned alignments
- Temporal saliency: Identify important time steps for predictions
### 12.4 Neuromorphic Implementations
Energy-efficient implementations:
- Spiking RNNs for neuromorphic hardware
- Low-precision arithmetic (4-8 bit)
- Event-driven processing for sparse temporal data
## 13. Summary & Key Takeaways
Core Concepts:
- LSTMs and GRUs solve vanishing gradient problem through additive memory paths and gating
- Bidirectional processing captures context from both directions
- Gradient clipping and careful initialization essential for stable training
Practical Considerations:
- LSTM hidden dimensions: 128-2048 depending on dataset size
- Recurrent dropout more effective than standard dropout
- Gradient clipping to norm 1-5 standard practice
- Bucketing sequences by length improves efficiency
Performance Insights:
- Attention mechanisms provide 20-40% improvement on sequence tasks
- Bidirectional encoding particularly beneficial for non-generative tasks
- Character-level features essential for rare word handling
Integration Strategy:
- LSTMs integrate naturally with attention, CNNs, and CRF layers
- Multi-layer architectures (2-4 layers) optimal for most tasks
- Residual connections enable deeper networks
Current Limitations:
- Inherently sequential inference prevents real-time parallelization
- Practical long-term dependency capture limited to ~200-300 steps
- Transformers have largely superseded RNNs for many NLP tasks
Despite transformer dominance in NLP, LSTMs and GRUs remain valuable for applications requiring strong inductive biases toward sequential processing, constrained computational budgets, and interpretable temporal dynamics.
---
## Appendix: Practical Implementation Labs
### Lab 1: Building an LSTM from Scratch
import numpy as np
class LSTM:
def __init__(self, input_size, hidden_size, output_size):
self.hidden_size = hidden_size
# Xavier initialization
scale = np.sqrt(1.0 / (input_size + hidden_size))
self.Wxh = np.random.randn(hidden_size, input_size) * scale
self.Whh = np.random.randn(hidden_size, hidden_size) * scale
self.Wxy = np.random.randn(output_size, hidden_size) * scale
self.bh = np.zeros(hidden_size)
self.by = np.zeros(output_size)
def forward(self, X):
T, D = X.shape
h = np.zeros(self.hidden_size)
c = np.zeros(self.hidden_size)
cache = []
for t in range(T):
x = X[t]
combined = np.concatenate([x, h])
# Simplified: single gate computation
i = 1.0 / (1.0 + np.exp(-(np.dot(self.Wxh, x) + np.dot(self.Whh, h))))
f = 1.0 / (1.0 + np.exp(-(np.dot(self.Wxh, x) + np.dot(self.Whh, h))))
c_tilde = np.tanh(np.dot(self.Wxh, x) + np.dot(self.Whh, h))
c = f * c + i * c_tilde
h = np.tanh(c)
cache.append((x, h, c))
y = np.dot(self.Wxy, h) + self.by
return y, cache
lstm = LSTM(input_size=10, hidden_size=20, output_size=5)
X = np.random.randn(50, 10) # 50 timesteps, 10D input
output, cache = lstm.forward(X)
print(f"Output shape: {output.shape}")### Lab 2: BiLSTM for Sequence Labeling
import torch
import torch.nn as nn
class BiLSTMTagger(nn.Module):
def __init__(self, vocab_size, embedding_dim, hidden_dim, num_tags):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=0)
self.bilstm = nn.LSTM(embedding_dim, hidden_dim,
batch_first=True, bidirectional=True)
self.classifier = nn.Linear(2 * hidden_dim, num_tags)
self.dropout = nn.Dropout(0.3)
def forward(self, tokens, lengths):
embedded = self.dropout(self.embedding(tokens))
packed = nn.utils.rnn.pack_padded_sequence(
embedded, lengths.cpu(), batch_first=True, enforce_sorted=False)
lstm_out, _ = self.bilstm(packed)
unpacked, _ = nn.utils.rnn.pad_packed_sequence(lstm_out, batch_first=True)
tags = self.classifier(unpacked)
return tags
model = BiLSTMTagger(vocab_size=1000, embedding_dim=100, hidden_dim=256, num_tags=10)
batch_tokens = torch.randint(0, 1000, (16, 50))
batch_lengths = torch.randint(30, 51, (16,))
output = model(batch_tokens, batch_lengths)
print(f"Tag predictions shape: {output.shape}") # [16, 50, 10]### Lab 3: Attention Mechanism with LSTM
import torch
import torch.nn as nn
import torch.nn.functional as F
class Attention(nn.Module):
def __init__(self, hidden_dim):
super().__init__()
self.query = nn.Linear(hidden_dim, hidden_dim)
self.key = nn.Linear(hidden_dim, hidden_dim)
self.value = nn.Linear(hidden_dim, hidden_dim)
self.scale = hidden_dim ** 0.5
def forward(self, decoder_state, encoder_states):
Q = self.query(decoder_state).unsqueeze(1) # [B, 1, H]
K = self.key(encoder_states) # [B, T, H]
V = self.value(encoder_states) # [B, T, H]
scores = torch.bmm(Q, K.transpose(1, 2)) / self.scale # [B, 1, T]
weights = F.softmax(scores, dim=-1)
context = torch.bmm(weights, V).squeeze(1) # [B, H]
return context, weights.squeeze(1)
attention = Attention(hidden_dim=512)
decoder_state = torch.randn(16, 512)
encoder_states = torch.randn(16, 50, 512)
context, weights = attention(decoder_state, encoder_states)
print(f"Context shape: {context.shape}, Attention weights shape: {weights.shape}")### Lab 4: GRU with Gradient Clipping
import torch
import torch.nn as nn
from torch.nn.utils import clip_grad_norm_
class GRUModel(nn.Module):
def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embedding_dim)
self.gru = nn.GRU(embedding_dim, hidden_dim, batch_first=True)
self.fc = nn.Linear(hidden_dim, output_dim)
self.dropout = nn.Dropout(0.2)
def forward(self, text, lengths):
embedded = self.dropout(self.embedding(text))
packed = nn.utils.rnn.pack_padded_sequence(
embedded, lengths.cpu(), batch_first=True, enforce_sorted=False)
_, hidden = self.gru(packed)
return self.fc(hidden.squeeze(0))
model = GRUModel(vocab_size=1000, embedding_dim=100, hidden_dim=256, output_dim=2)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()
# Training loop with gradient clipping
texts = torch.randint(0, 1000, (32, 100))
targets = torch.randint(0, 2, (32,))
lengths = torch.full((32,), 100)
outputs = model(texts, lengths)
loss = criterion(outputs, targets)
optimizer.zero_grad()
loss.backward()
clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
print(f"Loss: {loss.item():.4f}")