GRU Gated Recurrent Unit
# GRU: Gated Recurrent Unit
## Introduction & Motivation
GRU (Gated Recurrent Unit): simplified LSTM alternative. Reset gate: selective past information. Update gate: control information flow. Fewer parameters than LSTM; similar performance. Applications: sequence modeling, language tasks, time series.
Motivation: LSTM complex; GRU simpler with fewer parameters. Comparable performance on many tasks.
Applications: Sequential data, lightweight models.
---
## Core Concepts & Theory
### Reset Gate
Selective access to previous hidden state.
### Update Gate
Control information flow and state update.
### Candidate Hidden State
Computed with reset gate applied.
---
## Mathematical Formulation
Reset gate:
$$r_t = \sigma(W_r \cdot [h_{t-1}, x_t] + b_r)$$
Update gate:
$$z_t = \sigma(W_z \cdot [h_{t-1}, x_t] + b_z)$$
Candidate hidden state:
$$ ilde{h}_t = anh(W_h \cdot [r_t \odot h_{t-1}, x_t] + b_h)$$
Hidden state:
$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot ilde{h}_t$$
---
## Advanced Theory & Extensions
### Layered GRU
Stack multiple GRU layers; increased capacity.
### Bidirectional GRU
Process forward and backward; bidirectional context.
### GRU Variants
Simplified gates; parameter reduction.
---
## Computational Considerations
GRU: O(T · (3d² + 3d·x)) where T = time steps, d = hidden dim.
Parameter count: ~66% of LSTM; fewer parameters.
Memory: O(T · d) for storing hidden states.
---
## Practical Implementation Strategies
### Initialization
Xavier initialization; reset and update bias to 0.
### Bidirectional Processing
Forward and backward; concatenate outputs.
### Stacked Layers
Multiple GRU layers; deeper representations.
---
## Benchmark Datasets & Evaluation
Penn Treebank: Language modeling; comparison with LSTM.
Machine Translation: Competitive with LSTM on seq2seq.
Question Answering: Standard benchmark.
---
## Key Challenges & Limitations
### Sequence Length
Still limited by T; still O(T) dependencies.
### Gate Balance
Gates may collapse to constant; careful training.
### Comparison to LSTM
Task-dependent; often similar but not always better.
---
## Hyperparameter Tuning
Hidden dimension: 128-512; model capacity.
Dropout: 0.0-0.3; regularization.
Num layers: 1-3; stacking depth.
---
## Real-World Applications & Case Studies
Machine Translation: GRU encoder-decoder models.
Named Entity Recognition: Sequence labeling.
Sentiment Analysis: GRU + attention.
---
## Integration with Other Methods
GRU + Attention → sequence-to-sequence models.
GRU + CNN → hybrid architectures.
---
## Summary & Key Takeaways
GRU via reset and update gates provides simplified gating mechanism with fewer parameters than LSTM while maintaining comparable performance.
Principles:
1. Reset gate: selective history.
2. Update gate: state blending.
3. Candidate: reset-gated input.
4. Simpler: fewer parameters.
5. Comparable: similar performance.
---
---
## Appendix: Practical Labs
### Lab 1: GRU Cell Forward Pass
import numpy as np
class GRUCell:
def __init__(self, input_size, hidden_size):
self.input_size = input_size
self.hidden_size = hidden_size
# Initialize weights
self.W_r = np.random.randn(hidden_size, input_size + hidden_size) * 0.1
self.b_r = np.zeros((hidden_size, 1))
self.W_z = np.random.randn(hidden_size, input_size + hidden_size) * 0.1
self.b_z = np.zeros((hidden_size, 1))
self.W_h = np.random.randn(hidden_size, input_size + hidden_size) * 0.1
self.b_h = np.zeros((hidden_size, 1))
def forward(self, x, h_prev):
"""GRU cell forward pass"""
concat = np.vstack([h_prev, x])
# Reset gate
r_t = 1 / (1 + np.exp(-(np.dot(self.W_r, concat) + self.b_r)))
# Update gate
z_t = 1 / (1 + np.exp(-(np.dot(self.W_z, concat) + self.b_z)))
# Candidate hidden state
concat_r = np.vstack([r_t * h_prev, x])
h_tilde = np.tanh(np.dot(self.W_h, concat_r) + self.b_h)
# Hidden state
h_t = (1 - z_t) * h_prev + z_t * h_tilde
return h_t
# Test
np.random.seed(42)
gru = GRUCell(input_size=10, hidden_size=20)
x = np.random.randn(10, 1)
h_prev = np.zeros((20, 1))
h_t = gru.forward(x, h_prev)
assert h_t.shape == (20, 1), "Hidden shape"
print("✓ GRU cell forward working")
if __name__ == "__main__":
print("Lab 1: GRUCell - PASSED")### Lab 2: Reset and Update Gates
import numpy as np
def analyze_gru_gates(x, h_prev, weights):
"""Analyze GRU gate activations"""
concat = np.vstack([h_prev, x])
# Gate values
reset = 1 / (1 + np.exp(-np.dot(weights['W_r'], concat)))
update = 1 / (1 + np.exp(-np.dot(weights['W_z'], concat)))
return {
"reset_mean": reset.mean(),
"update_mean": update.mean(),
"reset_std": reset.std(),
"update_std": update.std(),
"reset_range": (reset.min(), reset.max()),
"update_range": (update.min(), update.max())
}
# Test
np.random.seed(42)
x = np.random.randn(10, 1)
h_prev = np.random.randn(20, 1)
weights = {
'W_r': np.random.randn(20, 30) * 0.1,
'W_z': np.random.randn(20, 30) * 0.1
}
gates = analyze_gru_gates(x, h_prev, weights)
assert all(0 <= gates['reset_mean'] <= 1), "Reset gate in [0,1]"
assert all(0 <= gates['update_mean'] <= 1), "Update gate in [0,1]"
print("✓ GRU gate analysis working")
if __name__ == "__main__":
print("Lab 2: GRUGateAnalysis - PASSED")### Lab 3: GRU vs LSTM Parameters
import numpy as np
def compare_rnn_parameters(hidden_size=256, input_size=128):
"""Compare parameters: RNN vs LSTM vs GRU"""
# Simple RNN: 1 matrix per step
rnn_params = hidden_size * (input_size + hidden_size) + hidden_size # W, U, b
# LSTM: 4 gates (forget, input, cell, output)
lstm_params = 4 * (hidden_size * (input_size + hidden_size) + hidden_size)
# GRU: 3 gates (reset, update, candidate)
gru_params = 3 * (hidden_size * (input_size + hidden_size) + hidden_size)
return {
"RNN": rnn_params,
"LSTM": lstm_params,
"GRU": gru_params,
"LSTM_vs_RNN": lstm_params / rnn_params,
"GRU_vs_RNN": gru_params / rnn_params,
"LSTM_vs_GRU": lstm_params / gru_params
}
# Test
params = compare_rnn_parameters(hidden_size=256, input_size=128)
assert params["LSTM"] > params["GRU"], "LSTM more params"
assert params["GRU"] > params["RNN"], "GRU more than RNN"
assert 3 < params["LSTM_vs_RNN"] < 5, "LSTM ~4x RNN"
assert 2 < params["GRU_vs_RNN"] < 4, "GRU ~3x RNN"
print("✓ RNN parameter comparison working")
if __name__ == "__main__":
print("Lab 3: ParameterComparison - PASSED")### Lab 4: GRU Sequence Processing
import numpy as np
def gru_sequence_forward(sequence, hidden_size=20):
"""Simple GRU sequence processing"""
T, input_size = sequence.shape
h_t = np.zeros((hidden_size, 1))
outputs = []
gates_log = {"reset": [], "update": []}
for t in range(T):
x_t = sequence[t:t+1].T # (input_size, 1)
concat = np.vstack([h_t, x_t])
# Simple gates (random for demo)
r_t = 1 / (1 + np.exp(-np.dot(np.random.randn(hidden_size, hidden_size + input_size), concat)))
z_t = 1 / (1 + np.exp(-np.dot(np.random.randn(hidden_size, hidden_size + input_size), concat)))
# Hidden update
h_tilde = np.tanh(np.dot(np.random.randn(hidden_size, hidden_size + input_size), concat))
h_t = (1 - z_t) * h_t + z_t * h_tilde
outputs.append(h_t.copy())
gates_log["reset"].append(r_t.mean())
gates_log["update"].append(z_t.mean())
return np.hstack(outputs), h_t, gates_log
# Test
np.random.seed(42)
sequence = np.random.randn(10, 5)
outputs, h_final, gates = gru_sequence_forward(sequence, hidden_size=20)
assert outputs.shape[1] == 10, "Sequence length"
assert len(gates["reset"]) == 10, "Gate logged"
print("✓ GRU sequence processing working")
if __name__ == "__main__":
print("Lab 4: GRUSequence - PASSED")