reinforcement learning fundamentals

# Reinforcement Learning Fundamentals

## Introduction & Motivation

RL: learn through interaction with environment. Maximize cumulative reward. Applications: game playing, robotics, control systems.

Motivation: Enable agents to learn optimal behaviors from experience.

Applications: Game playing, robotics, autonomous control.

---

## Core Concepts & Theory

### Markov Decision Process

States, actions, rewards, transitions.

### Value Function

Expected cumulative reward.

### Policy

Mapping from states to actions.

### Reward Signal

Feedback from environment.

---

## Mathematical Formulation

MDP:
$$M = (S, A, P, R, \gamma)$$

Value Function:
$$V(s) = \mathbb{E}[\sum_{t=0}^\infty \gamma^t R_t | s_0 = s]$$

Bellman Equation:
$$V(s) = R(s) + \gamma \mathbb{E}[V(s')]$$

---

## Advanced Theory & Extensions

### Exploration-Exploitation

Balance discovery and exploitation.

### Function Approximation

Use neural networks for V/Q.

### Experience Replay

Store and replay past experiences.

---

## Computational Considerations

Sample efficiency: Requires many interactions.

Computation: O(T·D²) per step.

Memory: Experience buffer.

---

## Practical Implementation Strategies

### Environment Interaction

Simulate or real world.

### Reward Design

Sparse or dense rewards.

### State Representation

Features or raw observations.

---

## Benchmark Datasets & Evaluation

Atari: Game playing benchmark.

MuJoCo: Continuous control.

OpenAI Gym: Standard environments.

---

## Key Challenges & Limitations

### Sample Inefficiency

Needs many interactions.

### Credit Assignment

Long-horizon problems.

### Non-stationarity

Changing policy.

---

## Hyperparameter Tuning

Discount factor: 0.95-0.99.

Learning rate: 1e-4 to 1e-3.

Exploration: ε-greedy, ε=0.1.

---

## Real-World Applications & Case Studies

Game Playing: AlphaGo, AlphaZero.

Robotics: Manipulation tasks.

Optimization: Resource allocation.

---

## Integration with Other Methods

RL + deep learning; + planning algorithms.

---

## Summary & Key Takeaways

RL learns from environmental interaction.

Principles:
1. MDP: Formal framework.
2. Value: Expected rewards.
3. Policy: Action selection.
4. Exploration: Discover good actions.
5. Learning: Iterative improvement.

---

## Appendix: Practical Labs

### Lab 1: MDP Setup

import numpy as np

class SimpleGridWorld:
 def __init__(self, grid_size=5):
 self.size = grid_size
 self.state = (0, 0)
 self.goal = (grid_size-1, grid_size-1)
 
 def step(self, action):
 # 0: up, 1: right, 2: down, 3: left
 moves = [(-1, 0), (0, 1), (1, 0), (0, -1)]
 dx, dy = moves[action]
 new_state = (self.state[0] + dx, self.state[1] + dy)
 
 # Boundary check
 new_state = (max(0, min(self.size-1, new_state[0])),
 max(0, min(self.size-1, new_state[1])))
 
 self.state = new_state
 
 reward = 1.0 if self.state == self.goal else -0.1
 done = self.state == self.goal
 
 return self.state, reward, done

env = SimpleGridWorld()
state, reward, done = env.step(1)
print(f"✓ MDP environment: state={state}, reward={reward}")

### Lab 2: Value Iteration

import numpy as np

def value_iteration(rewards, transitions, gamma=0.99, iterations=100):
 """Value iteration algorithm"""
 num_states = len(rewards)
 V = np.zeros(num_states)
 
 for _ in range(iterations):
 V_old = V.copy()
 
 for s in range(num_states):
 # Bellman update
 next_value = 0
 for s_next in range(num_states):
 next_value += transitions[s, s_next] * V_old[s_next]
 V[s] = rewards[s] + gamma * next_value
 
 return V

rewards = np.array([1.0, -0.1, -0.1, 10.0, -0.1])
transitions = np.random.dirichlet([1]*5, 5)
V = value_iteration(rewards, transitions)
print(f"✓ Value iteration: V={V}")

### Lab 3: Policy Evaluation

import numpy as np

def policy_evaluation(policy, rewards, transitions, gamma=0.99):
 """Evaluate policy"""
 num_states = len(rewards)
 V = np.zeros(num_states)
 
 for _ in range(10):
 for s in range(num_states):
 action = policy[s]
 next_value = np.sum(transitions[s] * V)
 V[s] = rewards[s] + gamma * next_value
 
 return V

policy = np.array([1, 0, 1, 0, 2]) # Actions per state
rewards = np.array([1.0, -0.1, -0.1, 10.0, -0.1])
transitions = np.random.dirichlet([1]*5, 5)
V = policy_evaluation(policy, rewards, transitions)
print(f"✓ Policy evaluation: V={V}")

### Lab 4: Q-Value Computation

import numpy as np

def compute_q_values(values, rewards, transitions, gamma=0.99):
 """Compute Q-values from V-values"""
 num_states = len(values)
 num_actions = 4
 Q = np.zeros((num_states, num_actions))
 
 for s in range(num_states):
 for a in range(num_actions):
 # Simplified: assume uniform action transitions
 next_value = values[s]
 Q[s, a] = rewards[s] + gamma * next_value
 
 return Q

values = np.random.rand(5)
rewards = np.array([1.0, -0.1, -0.1, 10.0, -0.1])
transitions = np.random.dirichlet([1]*5, 5)
Q = compute_q_values(values, rewards, transitions)
print(f"✓ Q-values computed: shape={Q.shape}")

---