Gated Recurrent Units GRU Simplified Gating
# Gated Recurrent Units: GRU & Simplified Gating
## Introduction & Motivation
GRU: simplified LSTM with two gates (reset, update). Fewer parameters; comparable performance. No separate cell state; hidden state carries all information. Applications: sequence modeling, faster training.
Motivation: LSTM powerful but complex (3 gates). GRU simplification reduces parameters while maintaining performance.
Applications: Language modeling, machine translation, time series forecasting.
---
## Core Concepts & Theory
### Reset Gate
Controls past hidden state contribution; selective memory reset.
### Update Gate
Controls blend of past hidden state and new candidate; balance.
### Candidate Hidden State
Linear combination of input and reset hidden state.
---
## Mathematical Formulation
GRU equations:
$$r_t = \sigma(W_r \cdot [h_{t-1}, x_t] + b_r)$$ (Reset gate)
$$z_t = \sigma(W_z \cdot [h_{t-1}, x_t] + b_z)$$ (Update gate)
$$ ilde{h}_t = anh(W \cdot [r_t * h_{t-1}, x_t] + b)$$ (Candidate)
$$h_t = (1 - z_t) * h_{t-1} + z_t * ilde{h}_t$$ (Hidden state update)
---
## Advanced Theory & Extensions
### Residual GRU
Add residual connections between layers.
### Bidirectional GRU
Process forward and backward; concatenate representations.
### Coupled Gates
Share weights between reset and update gates; further simplification.
---
## Computational Considerations
Per-timestep: O(D²) for matrix multiplies (2 gates vs LSTM 4).
Sequence: O(T × D²) for sequence length T.
Memory: O(T × D); no cell state overhead.
---
## Practical Implementation Strategies
### Batch Normalization on Recurrent Connections
Normalize hidden→hidden transformations.
### Layer Normalization
More stable than batch norm for RNNs.
### Regularization
Weight decay, dropout on recurrent connections.
---
## Benchmark Datasets & Evaluation
Penn Treebank: Comparable to LSTM (slightly worse, often acceptable).
Machine Translation: Similar BLEU scores as LSTM.
Time Series: Competitive with LSTM.
---
## Key Challenges & Limitations
### Simplified Dynamics
Fewer gates → less expressive (sometimes).
### Still Sequential
Not parallelizable; slower inference than Transformers.
### Long Sequences
Can still suffer gradient issues despite gating.
---
## Hyperparameter Tuning
Hidden size: 128-512; balance capacity-speed.
Number of layers: 1-3; similar to LSTM.
Dropout: 0.2-0.5 on recurrent connections.
---
## Real-World Applications & Case Studies
Machine Translation: Comparable BLEU to LSTM (faster).
Speech Recognition: Acoustic modeling.
Time Series: Forecasting, anomaly detection.
---
## Integration with Other Methods
GRU + Attention → Seq2Seq with attention (faster than LSTM).
GRU + Transformer Hybrid → combine strengths.
---
## Summary & Key Takeaways
GRU simplifies LSTM with reset and update gates, achieving competitive performance with fewer parameters and faster computation.
Principles:
1. Reset gate: selective memory reset; gated past state.
2. Update gate: blend past and new information; interpolation.
3. Candidate hidden state: nonlinear transformation.
4. Hidden state: direct update (no separate cell state).
5. Fewer parameters than LSTM; similar performance often.
---
---
## Appendix: Practical Labs
### Lab 1: GRU Cell
import torch
import torch.nn as nn
class GRUCell(nn.Module):
def __init__(self, input_size, hidden_size):
super().__init__()
self.input_size = input_size
self.hidden_size = hidden_size
# Reset, update, candidate gates
self.W_r = nn.Linear(input_size + hidden_size, hidden_size)
self.W_z = nn.Linear(input_size + hidden_size, hidden_size)
self.W_h = nn.Linear(input_size + hidden_size, hidden_size)
def forward(self, x, h):
combined = torch.cat([x, h], dim=1)
r = torch.sigmoid(self.W_r(combined)) # Reset
z = torch.sigmoid(self.W_z(combined)) # Update
combined_reset = torch.cat([x, r * h], dim=1)
h_tilde = torch.tanh(self.W_h(combined_reset))
h_new = (1 - z) * h + z * h_tilde
return h_new
# Test
cell = GRUCell(input_size=10, hidden_size=20)
x = torch.randn(5, 10)
h = torch.zeros(5, 20)
h_new = cell(x, h)
print(f"New hidden shape: {h_new.shape}")
assert h_new.shape == (5, 20), "Should match hidden size"
print("✓ GRU cell working")
if __name__ == "__main__":
print("Lab 1: Cell - PASSED")### Lab 2: GRU vs LSTM Parameters
import torch
import torch.nn as nn
def count_params(model):
"""Count trainable parameters"""
return sum(p.numel() for p in model.parameters() if p.requires_grad)
# Create LSTM and GRU with same hidden size
input_size, hidden_size, num_layers = 10, 20, 2
lstm = nn.LSTM(input_size, hidden_size, num_layers)
gru = nn.GRU(input_size, hidden_size, num_layers)
lstm_params = count_params(lstm)
gru_params = count_params(gru)
print(f"LSTM params: {lstm_params}, GRU params: {gru_params}")
print(f"GRU reduction: {(1 - gru_params/lstm_params)*100:.1f}%")
assert gru_params < lstm_params, "GRU should have fewer parameters"
print("✓ Parameter comparison working")
if __name__ == "__main__":
print("Lab 2: Parameters - PASSED")### Lab 3: Sequence with GRU
import torch
import torch.nn as nn
class GRUSequence(nn.Module):
def __init__(self, input_size, hidden_size, num_layers):
super().__init__()
self.gru = nn.GRU(input_size, hidden_size, num_layers, batch_first=True)
def forward(self, x):
output, h = self.gru(x)
return output, h
# Test
model = GRUSequence(input_size=10, hidden_size=32, num_layers=2)
x = torch.randn(8, 15, 10) # [B, T, D]
output, h = model(x)
print(f"Output shape: {output.shape}, Hidden shape: {h.shape}")
assert output.shape == (8, 15, 32), "Should output at each timestep"
assert h.shape == (2, 8, 32), "Hidden shape [num_layers, batch, hidden]"
print("✓ GRU sequence working")
if __name__ == "__main__":
print("Lab 3: Sequence - PASSED")### Lab 4: Training Dynamics
import torch
import torch.nn as nn
import numpy as np
def compare_gru_lstm_training(seq_len, input_size, hidden_size):
"""Compare training convergence"""
lstm = nn.LSTM(input_size, hidden_size, batch_first=True)
gru = nn.GRU(input_size, hidden_size, batch_first=True)
x = torch.randn(8, seq_len, input_size)
# LSTM forward pass
lstm_out, _ = lstm(x)
lstm_loss = lstm_out.sum()
lstm_loss.backward()
lstm_grad = x.grad.abs().mean() if x.grad is not None else 0
x.grad = None
# GRU forward pass
gru_out, _ = gru(x)
gru_loss = gru_out.sum()
gru_loss.backward()
gru_grad = x.grad.abs().mean() if x.grad is not None else 0
return lstm_grad.item() if isinstance(lstm_grad, torch.Tensor) else lstm_grad, gru_grad.item()
# Test
lstm_grad, gru_grad = compare_gru_lstm_training(seq_len=50, input_size=10, hidden_size=20)
print(f"LSTM gradient: {lstm_grad:.6f}, GRU gradient: {gru_grad:.6f}")
assert lstm_grad > 0 and gru_grad > 0, "Both should have positive gradients"
print("✓ Training dynamics comparison working")
if __name__ == "__main__":
print("Lab 4: Training - PASSED")