reinforcement learning basics
# Reinforcement Learning Basics
## Introduction & Motivation
RL: learn via interaction and rewards. Markov Decision Processes, value functions. Applications: game playing, robotics control.
Motivation: Learn policies from environmental feedback.
Applications: Autonomous agents, game AI.
---
## Core Concepts & Theory
### Markov Decision Process
State, action, reward, transition.
### Value Functions
Expected cumulative reward.
### Policy
Mapping states to actions.
### Bellman Equation
Recursive value decomposition.
---
## Mathematical Formulation
Bellman Equation:
$$V(s) = \mathbb{E}[r + \gamma V(s')]$$
Policy Evaluation:
$$V^{\pi}(s) = \mathbb{E}[r + \gamma V^{\pi}(s') | \pi]$$
Optimal Value:
$$V^*(s) = \max_a \mathbb{E}[r + \gamma V^*(s')]$$
---
## Advanced Theory & Extensions
### Temporal Difference Learning
Bootstrap from next state.
### Monte Carlo Methods
Trajectory-based learning.
### Function Approximation
Neural network value estimates.
---
## Computational Considerations
Value iteration: O(S² A).
Policy evaluation: O(S² A).
TD update: O(1).
---
## Practical Implementation Strategies
### Experience Replay
Store and sample transitions.
### Target Networks
Separate evaluation and target.
### Exploration Strategies
Epsilon-greedy and upper confidence bound.
---
## Benchmark Environments
OpenAI Gym: Standardized benchmarks.
Atari: Classic game playing.
MuJoCo: Continuous control.
---
## Key Challenges & Limitations
### Sample Efficiency
High sample complexity.
### Exploration-Exploitation
Balancing trade-off.
### Non-Stationarity
Changing value targets.
---
## Hyperparameter Tuning
Discount factor (gamma): 0.95-0.99.
Learning rate: 1e-4 to 1e-2.
Epsilon (exploration): 0.01-0.1.
---
## Real-World Applications & Case Studies
Game Playing: Atari and board games.
Robot Control: Continuous action spaces.
Autonomous Driving: Path planning.
---
## Integration with Other Methods
RL + deep learning for function approximation; + imitation learning for initialization.
---
## Summary & Key Takeaways
Reinforcement Learning learns policies via interaction with environments.
Principles:
1. MDP: Problem formulation.
2. Value function: Expected return estimation.
3. Policy: Action selection strategy.
4. Bellman: Recursive decomposition.
5. TD learning: Bootstrap-based updates.
---
## Appendix: Practical Labs
### Lab 1: Bellman Equation Evaluation
import numpy as np
def bellman_update(V, s, r, s_prime, gamma=0.99):
"""Update value function using Bellman equation"""
V[s] = r + gamma * V[s_prime]
return V
np.random.seed(42)
V = np.random.randn(10)
r = 1.0
s, s_prime = 3, 5
gamma = 0.99
V = bellman_update(V, s, r, s_prime, gamma)
assert V[s] > 0, "Value updated"
print("✓ Bellman update working")### Lab 2: Policy Evaluation
import numpy as np
def policy_evaluation(P, R, policy, gamma=0.99, iterations=10):
"""Evaluate a policy via iterative value updates"""
num_states = len(policy)
V = np.zeros(num_states)
for _ in range(iterations):
V_new = np.zeros(num_states)
for s in range(num_states):
a = policy[s]
V_new[s] = R[s, a] + gamma * sum(P[s, a, s_prime] * V[s_prime] for s_prime in range(num_states))
V = V_new
return V
np.random.seed(42)
num_states = 5
P = np.random.dirichlet(np.ones(num_states), size=(num_states, 2))
R = np.random.randn(num_states, 2)
policy = np.array([0, 1, 0, 1, 0])
V = policy_evaluation(P, R, policy)
assert V.shape == (5,), "Correct value shape"
print("✓ Policy evaluation working")### Lab 3: Epsilon-Greedy Exploration
import numpy as np
def epsilon_greedy(Q, epsilon=0.1):
"""Select action using epsilon-greedy strategy"""
if np.random.rand() < epsilon:
action = np.random.randint(Q.shape[0])
else:
action = np.argmax(Q)
return action
np.random.seed(42)
Q = np.random.randn(5)
action = epsilon_greedy(Q, epsilon=0.1)
assert 0 <= action < 5, "Valid action"
print("✓ Epsilon-greedy exploration working")### Lab 4: Experience Replay Buffer
import numpy as np
class ReplayBuffer:
def __init__(self, max_size=1000):
self.max_size = max_size
self.buffer = []
def add(self, transition):
if len(self.buffer) >= self.max_size:
self.buffer.pop(0)
self.buffer.append(transition)
def sample(self, batch_size):
indices = np.random.choice(len(self.buffer), batch_size)
return [self.buffer[i] for i in indices]
buffer = ReplayBuffer(max_size=100)
for i in range(50):
buffer.add((i, i+1, 1, i+2))
batch = buffer.sample(10)
assert len(batch) == 10, "Correct batch size"
print("✓ Experience replay buffer working")---