Reinforcement Learning Basics

# Reinforcement Learning Basics

## Introduction & Motivation

RL: learn via interaction and rewards. Markov Decision Processes, value functions. Applications: game playing, robotics control.

Motivation: Learn policies from environmental feedback.

Applications: Autonomous agents, game AI.

---

## Core Concepts & Theory

### Markov Decision Process

State, action, reward, transition.

### Value Functions

Expected cumulative reward.

### Policy

Mapping states to actions.

### Bellman Equation

Recursive value decomposition.

---

## Mathematical Formulation

Bellman Equation:
$$V(s) = \mathbb{E}[r + \gamma V(s')]$$

Policy Evaluation:
$$V^{\pi}(s) = \mathbb{E}[r + \gamma V^{\pi}(s') | \pi]$$

Optimal Value:
$$V^*(s) = \max_a \mathbb{E}[r + \gamma V^*(s')]$$

---

## Advanced Theory & Extensions

### Temporal Difference Learning

Bootstrap from next state.

### Monte Carlo Methods

Trajectory-based learning.

### Function Approximation

Neural network value estimates.

---

## Computational Considerations

Value iteration: O(S² A).

Policy evaluation: O(S² A).

TD update: O(1).

---

## Practical Implementation Strategies

### Experience Replay

Store and sample transitions.

### Target Networks

Separate evaluation and target.

### Exploration Strategies

Epsilon-greedy and upper confidence bound.

---

## Benchmark Environments

OpenAI Gym: Standardized benchmarks.

Atari: Classic game playing.

MuJoCo: Continuous control.

---

## Key Challenges & Limitations

### Sample Efficiency

High sample complexity.

### Exploration-Exploitation

Balancing trade-off.

### Non-Stationarity

Changing value targets.

---

## Hyperparameter Tuning

Discount factor (gamma): 0.95-0.99.

Learning rate: 1e-4 to 1e-2.

Epsilon (exploration): 0.01-0.1.

---

## Real-World Applications & Case Studies

Game Playing: Atari and board games.

Robot Control: Continuous action spaces.

Autonomous Driving: Path planning.

---

## Integration with Other Methods

RL + deep learning for function approximation; + imitation learning for initialization.

---

## Summary & Key Takeaways

Reinforcement Learning learns policies via interaction with environments.

Principles:
1. MDP: Problem formulation.
2. Value function: Expected return estimation.
3. Policy: Action selection strategy.
4. Bellman: Recursive decomposition.
5. TD learning: Bootstrap-based updates.

---

## Appendix: Practical Labs

### Lab 1: Bellman Equation Evaluation

import numpy as np

def bellman_update(V, s, r, s_prime, gamma=0.99):
 """Update value function using Bellman equation"""
 V[s] = r + gamma * V[s_prime]
 return V

np.random.seed(42)
V = np.random.randn(10)
r = 1.0
s, s_prime = 3, 5
gamma = 0.99
V = bellman_update(V, s, r, s_prime, gamma)
assert V[s] > 0, "Value updated"
print("✓ Bellman update working")

### Lab 2: Policy Evaluation

import numpy as np

def policy_evaluation(P, R, policy, gamma=0.99, iterations=10):
 """Evaluate a policy via iterative value updates"""
 num_states = len(policy)
 V = np.zeros(num_states)
 
 for _ in range(iterations):
 V_new = np.zeros(num_states)
 for s in range(num_states):
 a = policy[s]
 V_new[s] = R[s, a] + gamma * sum(P[s, a, s_prime] * V[s_prime] for s_prime in range(num_states))
 V = V_new
 
 return V

np.random.seed(42)
num_states = 5
P = np.random.dirichlet(np.ones(num_states), size=(num_states, 2))
R = np.random.randn(num_states, 2)
policy = np.array([0, 1, 0, 1, 0])
V = policy_evaluation(P, R, policy)
assert V.shape == (5,), "Correct value shape"
print("✓ Policy evaluation working")

### Lab 3: Epsilon-Greedy Exploration

import numpy as np

def epsilon_greedy(Q, epsilon=0.1):
 """Select action using epsilon-greedy strategy"""
 if np.random.rand() < epsilon:
 action = np.random.randint(Q.shape[0])
 else:
 action = np.argmax(Q)
 return action

np.random.seed(42)
Q = np.random.randn(5)
action = epsilon_greedy(Q, epsilon=0.1)
assert 0 <= action < 5, "Valid action"
print("✓ Epsilon-greedy exploration working")

### Lab 4: Experience Replay Buffer

import numpy as np

class ReplayBuffer:
 def __init__(self, max_size=1000):
 self.max_size = max_size
 self.buffer = []
 
 def add(self, transition):
 if len(self.buffer) >= self.max_size:
 self.buffer.pop(0)
 self.buffer.append(transition)
 
 def sample(self, batch_size):
 indices = np.random.choice(len(self.buffer), batch_size)
 return [self.buffer[i] for i in indices]

buffer = ReplayBuffer(max_size=100)
for i in range(50):
 buffer.add((i, i+1, 1, i+2))
 
batch = buffer.sample(10)
assert len(batch) == 10, "Correct batch size"
print("✓ Experience replay buffer working")

---