Reinforcement Learning Q-Learning Value-Based Methods
# Reinforcement Learning: Q-Learning & Value-Based Methods
## Introduction & Motivation
Q-Learning: learn value function via temporal difference. Off-policy; model-free. Q-networks; deep Q-learning. Applications: game playing, robotics, control.
Motivation: Learn optimal policy from interactions.
Applications: Games, robotics, sequential decision-making.
---
## Core Concepts & Theory
### Value Function
Expected return from state.
### Q-Function
State-action value; expected reward.
### Temporal Difference
Bootstrap from next state value.
---
## Mathematical Formulation
Bellman equation:
$$V(s) = \max_a \mathbb{E}[r + \gamma V(s')]$$
Q-Learning update:
$$Q(s,a) \leftarrow Q(s,a) + \alpha[r + \gamma \max_{a'} Q(s', a') - Q(s,a)]$$
Deep Q-Network loss:
$$L = \mathbb{E}[(r + \gamma \max_{a'} Q(s', a'; heta^-) - Q(s, a; heta))^2]$$
---
## Advanced Theory & Extensions
### Double DQN
Reduce overestimation bias.
### Dueling DQN
Separate value and advantage.
### Prioritized Experience Replay
Sample important transitions.
---
## Computational Considerations
Q-table: O(|S|·|A|) space.
DQN: O(batch_size·model_size).
Exploration: O(epsilon-greedy).
---
## Practical Implementation Strategies
### Epsilon-Greedy
Exploration-exploitation trade-off.
### Experience Replay
Break correlations; stability.
### Target Network
Separate networks for stability.
---
## Benchmark Environments
Atari: Game-playing benchmark.
CartPole: Control benchmark.
MuJoCo: Continuous control.
---
## Key Challenges & Limitations
### Overestimation
Max operator bias.
### Sample Efficiency
Requires many interactions.
### Convergence
Unstable training.
---
## Hyperparameter Tuning
Learning rate α: 1e-4 to 1e-3.
Discount γ: 0.99 typical.
Epsilon decay: Gradual reduction.
---
## Real-World Applications & Case Studies
Game AI: Atari mastery.
Robotics: Control policies.
Autonomous Driving: Decision making.
---
## Integration with Other Methods
Q-Learning + Policy Gradient → actor-critic.
Q-Learning + Exploration → active RL.
---
## Summary & Key Takeaways
Q-Learning via temporal difference enables optimal policy discovery through value function approximation and bootstrapping.
Principles:
1. Value: expected return.
2. Q-function: state-action value.
3. Temporal difference: bootstrapping.
4. Off-policy: behavior policy.
5. DQN: neural approximation.
---
---
## Appendix: Practical Labs
### Lab 1: Q-Learning Update
import numpy as np
def q_learning_update(Q, state, action, reward, next_state, alpha=0.1, gamma=0.99):
"""Q-Learning update rule"""
if next_state is None:
target = reward
else:
target = reward + gamma * np.max(Q[next_state])
td_error = target - Q[state, action]
Q[state, action] += alpha * td_error
return Q
# Test
np.random.seed(42)
Q = np.random.randn(10, 4)
s, a, r, s_next = 0, 1, 1.0, 2
Q_updated = q_learning_update(Q, s, a, r, s_next)
assert Q_updated.shape == Q.shape, "Q shape"
print("✓ Q-Learning working")
if __name__ == "__main__":
print("Lab 1: QLearning - PASSED")### Lab 2: Epsilon-Greedy Policy
import numpy as np
def epsilon_greedy_action(Q, state, epsilon=0.1, num_actions=4):
"""Select action with epsilon-greedy"""
if np.random.rand() < epsilon:
return np.random.randint(0, num_actions)
else:
return np.argmax(Q[state])
# Test
np.random.seed(42)
Q = np.random.randn(10, 4)
state = 0
action = epsilon_greedy_action(Q, state, epsilon=0.1)
assert 0 <= action < 4, "Valid action"
print("✓ Epsilon-greedy working")
if __name__ == "__main__":
print("Lab 2: EpsilonGreedy - PASSED")### Lab 3: Experience Replay
import numpy as np
def sample_batch_from_replay_buffer(buffer, batch_size=32):
"""Sample batch from experience replay"""
indices = np.random.choice(len(buffer), batch_size, replace=False)
batch = [buffer[i] for i in indices]
return batch
# Test
np.random.seed(42)
buffer = [(np.random.randn(4), i, 1.0, np.random.randn(4), False) for i in range(1000)]
batch = sample_batch_from_replay_buffer(buffer, batch_size=32)
assert len(batch) == 32, "Batch size"
print("✓ Experience replay working")
if __name__ == "__main__":
print("Lab 3: ExperienceReplay - PASSED")### Lab 4: DQN Loss
import numpy as np
def dqn_loss(Q_current, Q_target, rewards, next_states, dones, gamma=0.99):
"""Deep Q-Network loss"""
batch_size = len(rewards)
# Target Q-values
targets = rewards.copy()
for i in range(batch_size):
if not dones[i]:
targets[i] += gamma * np.max(Q_target[next_states[i]])
# Predictions
predictions = Q_current
# MSE loss
loss = np.mean((predictions - targets) ** 2)
return loss
# Test
np.random.seed(42)
Q_curr = np.random.randn(32, 4)
Q_targ = np.random.randn(32, 4)
rewards = np.random.rand(32)
next_states = np.arange(32)
dones = np.random.rand(32) > 0.5
loss = dqn_loss(Q_curr, Q_targ, rewards, next_states, dones)
assert np.isfinite(loss), "Loss finite"
print("✓ DQN loss working")
if __name__ == "__main__":
print("Lab 4: DQNLoss - PASSED")