Policy Gradient Methods

# Policy Gradient Methods

## Introduction & Motivation

Policy gradient: directly optimize policy through gradient ascent. Learn action selection. Applications: continuous control, game playing.

Motivation: Directly optimize policy parameters for better actions.

Applications: Robotics control, game playing, optimization.

---

## Core Concepts & Theory

### Policy Gradient Theorem

Gradient of expected return.

### REINFORCE

Basic policy gradient algorithm.

### Advantage Functions

Reduce variance in gradients.

### Baseline Subtraction

Variance reduction technique.

---

## Mathematical Formulation

Policy Gradient:
$$ abla_ heta J( heta) = \mathbb{E}[∇_ heta \log π_ heta(a|s) Q(s,a)]$$

REINFORCE:
$$ abla_ heta J = \mathbb{E}[∇_ heta \log π_ heta(a|s) R_t]$$

With Baseline:
$$ abla_ heta J = \mathbb{E}[∇_ heta \log π_ heta(a|s) (R_t - b(s))]$$

---

## Advanced Theory & Extensions

### Actor-Critic

Combine policy and value.

### Natural Gradient

Fisher information matrix.

### Trust Region

Constrain policy updates.

---

## Computational Considerations

Gradient computation: O(T·D).

Variance: High without baselines.

Sample efficiency: Improves with advantage.

---

## Practical Implementation Strategies

### Advantage Estimation

GAE or n-step returns.

### Baseline Network

Value function approximation.

### Entropy Regularization

Encourage exploration.

---

## Benchmark Datasets & Evaluation

MuJoCo: Continuous control.

Atari: Discrete actions.

OpenAI Gym: Standard environments.

---

## Key Challenges & Limitations

### Variance

Policy gradient gradients noisy.

### Sample Efficiency

Requires many episodes.

### Local Optima

May converge to suboptimal.

---

## Hyperparameter Tuning

Learning rate: 1e-4 to 1e-3.

Discount factor: 0.95-0.99.

Entropy coefficient: 0.01-0.1.

---

## Real-World Applications & Case Studies

Robotics: Control policies.

Game AI: Atari, Go.

Optimization: Resource allocation.

---

## Integration with Other Methods

Policy gradient + value baseline; + trust region constraints.

---

## Summary & Key Takeaways

Policy gradient methods directly optimize policies.

Principles:
1. Gradient ascent: Maximize return.
2. Log derivative: Efficient gradient.
3. Variance: Reduce with baseline.
4. Advantage: Reduce gradient noise.
5. Exploration: Add entropy bonus.

---

## Appendix: Practical Labs

### Lab 1: REINFORCE Update

import numpy as np

def reinforce_update(log_probs, rewards, learning_rate=0.01):
 """REINFORCE policy gradient update"""
 gradient = 0
 discounted_reward = 0
 
 for t in reversed(range(len(rewards))):
 discounted_reward = rewards[t] + 0.99 * discounted_reward
 gradient += log_probs[t] * discounted_reward
 
 policy_gradient = gradient / len(rewards)
 return policy_gradient

log_probs = np.array([np.log(0.8), np.log(0.7), np.log(0.6)])
rewards = np.array([1.0, 0.5, 0.2])
grad = reinforce_update(log_probs, rewards)
print(f"✓ REINFORCE gradient: {grad:.3f}")

### Lab 2: Advantage Computation

import numpy as np

def compute_advantages(rewards, values, gamma=0.99, gae_lambda=0.95):
 """Generalized Advantage Estimation"""
 advantages = []
 advantage = 0
 
 for t in reversed(range(len(rewards))):
 td_error = rewards[t] - values[t]
 if t < len(rewards) - 1:
 td_error += gamma * values[t + 1]
 
 advantage = td_error + gamma * gae_lambda * advantage
 advantages.insert(0, advantage)
 
 return np.array(advantages)

rewards = np.array([1.0, 0.5, 0.2, 0.0])
values = np.array([0.8, 0.4, 0.1, 0.0])
adv = compute_advantages(rewards, values)
print(f"✓ Advantages: {adv}")

### Lab 3: Entropy Regularization

import numpy as np

def compute_entropy(probs):
 """Compute policy entropy"""
 entropy = -np.sum(probs * np.log(probs + 1e-8))
 return entropy

def policy_gradient_with_entropy(log_probs, advantages, entropy_coeff=0.01):
 """Policy gradient with entropy bonus"""
 pg_loss = -np.mean(log_probs * advantages)
 
 # Entropy bonus (maximize exploration)
 entropy = compute_entropy(np.exp(log_probs))
 
 total_loss = pg_loss - entropy_coeff * entropy
 return total_loss

np.random.seed(42)
log_probs = np.random.randn(10)
advantages = np.random.randn(10)
loss = policy_gradient_with_entropy(log_probs, advantages)
print(f"✓ Policy loss with entropy: {loss:.3f}")

### Lab 4: Baseline Variance Reduction

import numpy as np

def compare_with_without_baseline(returns, baseline):
 """Compare gradient variance with/without baseline"""
 # Without baseline
 var_without = np.var(returns)
 
 # With baseline
 advantages = returns - baseline
 var_with = np.var(advantages)
 
 variance_reduction = 1 - var_with / var_without
 return variance_reduction

returns = np.array([10.5, 9.8, 10.2, 10.1, 9.9])
baseline = np.mean(returns)
reduction = compare_with_without_baseline(returns, baseline)
assert reduction > 0
print(f"✓ Variance reduction: {reduction:.1%}")

---