Reinforcement Learning Policy Gradient Reinforce
# Reinforcement Learning: Policy Gradient & REINFORCE
## Introduction & Motivation
Policy gradient: optimize policy directly via gradient. REINFORCE: likelihood ratio gradient estimator. Actor-only: no value function baseline. Reducing variance: baseline subtraction; important sampling. Applications: continuous control, game playing, robotics.
Motivation: Q-learning: off-policy, discrete. Policy gradient: on-policy, continuous actions.
Applications: Continuous control, game AI, robotics.
---
## Core Concepts & Theory
### Policy Gradient Theorem
∇J(θ) ∝ E[∇ log π(a|s) Q(s,a)].
### REINFORCE
Sample trajectories; compute returns; gradient estimate.
### Baseline
Subtract value function; reduce variance without bias.
---
## Mathematical Formulation
Policy Gradient:
$$
abla J( heta) = \mathbb{E}[
abla_ heta \log \pi_ heta(a|s) Q(s,a)]$$
REINFORCE gradient:
$$
abla J( heta) = \mathbb{E}[
abla_ heta \log \pi_ heta(a|s) G_t]$$
where G_t = discounted return.
With baseline:
$$
abla J( heta) = \mathbb{E}[
abla_ heta \log \pi_ heta(a|s) (G_t - V(s))]$$
---
## Advanced Theory & Extensions
### Actor-Critic
Separate actor (policy) and critic (value); reduce variance.
### PPO (Proximal Policy Optimization)
Clip objective; stable policy updates.
### Trust Region Policy Optimization (TRPO)
Natural gradient + trust region; theoretical guarantees.
---
## Computational Considerations
Sampling: O(T·env_steps) per trajectory.
Gradient: O(T·network_size) per batch.
Training: Typically T=1000+ trajectories.
---
## Practical Implementation Strategies
### Return Computation
Discounted sum; γ=0.99 typical.
### Advantage Estimation
Generalized advantage: λ=0.95 common.
### Learning Rate
Lower than supervised; 3e-4 common.
---
## Benchmark Datasets & Evaluation
MuJoCo: Continuous control; average return metric.
Atari: Discrete actions; score metric.
Robotic Manipulation: real-world sim2real.
---
## Key Challenges & Limitations
### High Variance
Sample trajectories; noisy gradient. Importance sampling helps.
### Sample Efficiency
On-policy; needs many samples. Off-policy methods better.
### Convergence
Local optima; no guarantees.
---
## Hyperparameter Tuning
Learning rate: 1e-4 to 1e-3; lower for stability.
Batch size: 32-128 trajectories; larger = more stable.
Discount γ: 0.99 standard; 0.95 for short-term.
---
## Real-World Applications & Case Studies
Game Playing: Policy gradient in Atari.
Robotics: Continuous control via PPO, SAC.
Autonomous Driving: End-to-end learning.
---
## Integration with Other Methods
Policy Gradient + Value Function → Actor-Critic.
Policy Gradient + Importance Sampling → off-policy.
---
## Summary & Key Takeaways
Policy gradient via REINFORCE and actor-critic methods optimize policies directly through gradient ascent on expected returns, enabling continuous control and flexible action spaces.
Principles:
1. REINFORCE: likelihood ratio gradient.
2. Baseline: reduce variance without bias.
3. Actor-critic: combine policy and value.
4. Advantage: G_t - V(s); reduce variance.
5. PPO: clipped objective; stable updates.
---
---
## Appendix: Practical Labs
### Lab 1: REINFORCE Algorithm
import torch
import numpy as np
def compute_returns(rewards, gamma=0.99):
"""Compute discounted returns"""
returns = []
G = 0
for r in reversed(rewards):
G = r + gamma * G
returns.insert(0, G)
return torch.tensor(returns, dtype=torch.float32)
# Test
np.random.seed(42)
rewards = [1.0, 2.0, 1.5, 0.5]
returns = compute_returns(rewards, gamma=0.99)
assert returns.shape == (4,), "Returns shape correct"
assert returns[0] > returns[-1], "Discounting reduces"
print("✓ Return computation working")
if __name__ == "__main__":
print("Lab 1: Returns - PASSED")### Lab 2: Policy Gradient Loss
import torch
import torch.nn as nn
import numpy as np
def policy_gradient_loss(log_probs, advantages):
"""REINFORCE loss"""
loss = -(log_probs * advantages).mean()
return loss
# Test
np.random.seed(42)
log_probs = torch.log(torch.rand(32) + 1e-8)
advantages = torch.randn(32)
loss = policy_gradient_loss(log_probs, advantages)
assert torch.isfinite(loss), "Loss finite"
assert loss >= 0, "Loss non-negative"
print("✓ Policy gradient loss working")
if __name__ == "__main__":
print("Lab 2: Loss - PASSED")### Lab 3: Actor-Critic
import torch
import torch.nn as nn
import numpy as np
class ActorCritic(nn.Module):
def __init__(self, state_dim, action_dim, hidden_dim=64):
super().__init__()
self.shared = nn.Sequential(
nn.Linear(state_dim, hidden_dim),
nn.ReLU()
)
self.actor = nn.Linear(hidden_dim, action_dim)
self.critic = nn.Linear(hidden_dim, 1)
def forward(self, state):
feat = self.shared(state)
action_logits = self.actor(feat)
value = self.critic(feat)
return action_logits, value
# Test
np.random.seed(42)
model = ActorCritic(state_dim=10, action_dim=4)
state = torch.randn(8, 10)
action_logits, value = model(state)
assert action_logits.shape == (8, 4), "Action logits shape"
assert value.shape == (8, 1), "Value shape"
print("✓ Actor-critic working")
if __name__ == "__main__":
print("Lab 3: ActorCritic - PASSED")### Lab 4: Advantage Computation
import numpy as np
def compute_advantages(rewards, values, gamma=0.99, lam=0.95):
"""Generalized advantage estimation"""
advantages = []
gae = 0
for t in range(len(rewards) - 1, -1, -1):
if t < len(rewards) - 1:
delta = rewards[t] + gamma * values[t+1] - values[t]
else:
delta = rewards[t] - values[t]
gae = delta + gamma * lam * gae
advantages.insert(0, gae)
return np.array(advantages)
# Test
np.random.seed(42)
rewards = [1.0, 2.0, 1.5, 0.5]
values = [0.5, 1.0, 1.2, 0.3]
advantages = compute_advantages(rewards, values)
assert len(advantages) == 4, "Advantages length"
assert np.isfinite(advantages).all(), "All finite"
print("✓ Advantage computation working")
if __name__ == "__main__":
print("Lab 4: Advantages - PASSED")