Policy Gradient Methods Actor-Critic Advantage Functions
# Policy Gradient Methods: Actor-Critic & Advantage Functions
## Introduction & Motivation
Policy gradient methods directly optimize policy via gradient ascent. REINFORCE uses Monte Carlo returns; reduces variance via baseline (advantage). Actor-critic separates policy (actor) and value (critic) networks. Enables continuous action spaces, off-policy learning variants (A3C, PPO, TRPO).
Motivation: Q-learning requires tabular or function approximation for discrete actions. Policy gradients naturally handle continuous spaces via differentiable policy.
Applications: Continuous control, game playing (Atari), robotics, resource allocation.
---
## Core Concepts & Theory
### Policy Gradient Theorem
∇J(θ) ∝ E[∇log π_θ(a|s) Q^π(s,a)].
Gradient of objective = expected gradient of log-policy weighted by action value.
### Advantage Function
A(s,a) = Q(s,a) - V(s).
Measures relative value of action a in state s.
### Actor-Critic
Actor π_θ(a|s): updates via policy gradient.
Critic V_φ(s): estimates baseline via TD error.
---
## Mathematical Formulation
Policy gradient objective:
$$J( heta) = \mathbb{E}_{s \sim p^\pi, a \sim \pi_ heta}[\log \pi_ heta(a|s) Q^\pi(s,a)]$$
REINFORCE update:
$$ heta \leftarrow heta + \alpha
abla \log \pi_ heta(a|s) G_t$$
Advantage-weighted update:
$$ heta \leftarrow heta + \alpha
abla \log \pi_ heta(a|s) (G_t - V_\phi(s))$$
---
## Advanced Theory & Extensions
### Generalized Advantage Estimation (GAE)
Λ_t = δ_t + (γλ)δ_{t+1} + (γλ)²δ_{t+2} + ...
Bias-variance tradeoff via λ ∈ [0,1].
### Trust Region Policy Optimization (TRPO)
Constrain KL divergence between old/new policy. Ensures stability.
### Proximal Policy Optimization (PPO)
Simpler; clip probability ratio. Widely adopted.
---
## Computational Considerations
Trajectory collection: O(T × |state|) per episode.
Gradient computation: O(T × |θ|) backprop.
Parallel workers: A3C uses multiple threads; wall-clock speedup.
---
## Practical Implementation Strategies
### Standardizing Advantages
(A - mean(A)) / (std(A) + ε). Reduces gradient variance.
### Entropy Regularization
Add -β H(π) to objective; encourages exploration.
### Learning Rate Scheduling
Decay or adaptive (Adam); typical 3e-4 to 1e-3.
---
## Benchmark Environments & Evaluation
MuJoCo: Continuous control; humanoid, walker.
Atari: Discrete actions; visual learning.
Metrics: Episode return, sample efficiency, convergence speed.
---
## Key Challenges & Limitations
### High Variance
Trajectories are noisy; many samples needed. Baselines and GAE reduce variance.
### Off-Policy Corrections
Importance sampling leads to high variance. On-policy better but sample-inefficient.
### Non-Stationary Rewards
Reward distribution shifts during training; normalization helps.
---
## Hyperparameter Tuning
Learning rate: 1e-4 to 1e-3.
Entropy bonus β: 0.01 to 0.1.
GAE λ: 0.95-0.99.
Batch size: 32-256.
---
## Real-World Applications & Case Studies
AlphaStar: RL + supervised learning; masters StarCraft II.
Robotics: Continuous control; sim-to-real transfer via domain randomization.
Recommendation Systems: Slate generation via policy gradient.
---
## Integration with Other Methods
Policy Gradient + Model Learning → world models, planning.
Policy Gradient + Curiosity → intrinsic motivation, exploration.
---
## Summary & Key Takeaways
Policy gradient methods directly optimize policies via gradient ascent, with actor-critic architecture reducing variance and enabling efficient learning.
Principles:
1. Policy gradient theorem: ∇J ∝ E[∇log π Q(s,a)].
2. Advantage baseline reduces variance without bias.
3. Actor-critic separates policy and value learning.
4. Entropy regularization encourages exploration.
5. GAE balances bias-variance via λ.
---
---
## Appendix: Practical Labs
### Lab 1: REINFORCE on CartPole
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import gym
class PolicyNetwork(nn.Module):
def __init__(self, state_dim=4, action_dim=2):
super().__init__()
self.fc = nn.Sequential(
nn.Linear(state_dim, 128),
nn.ReLU(),
nn.Linear(128, action_dim)
)
def forward(self, state):
return torch.softmax(self.fc(state), dim=-1)
env = gym.make('CartPole-v1')
policy = PolicyNetwork(state_dim=4, action_dim=2)
optimizer = optim.Adam(policy.parameters(), lr=1e-3)
gamma = 0.99
returns = []
for episode in range(50):
state, _ = env.reset()
log_probs, rewards = [], []
for step in range(200):
state_tensor = torch.FloatTensor(state).unsqueeze(0)
probs = policy(state_tensor)
dist = torch.distributions.Categorical(probs)
action = dist.sample()
log_probs.append(dist.log_prob(action))
state, reward, terminated, truncated, _ = env.step(action.item())
rewards.append(reward)
if terminated or truncated:
break
G = 0
loss = 0
for t in reversed(range(len(rewards))):
G = rewards[t] + gamma * G
loss -= log_probs[t] * G
optimizer.zero_grad()
loss.backward()
optimizer.step()
ep_return = sum(rewards)
returns.append(ep_return)
avg_return = np.mean(returns[-10:])
print(f"Average return (last 10 eps): {avg_return:.2f}")
assert avg_return > 50, "Should learn to balance pole"
assert len(returns) == 50, "Should have 50 episodes"
print("✓ REINFORCE working")
if __name__ == "__main__":
print("Lab 1: REINFORCE - PASSED")### Lab 2: Advantage Baseline
import torch
import torch.nn as nn
import numpy as np
class ActorCritic(nn.Module):
def __init__(self, state_dim=4, action_dim=2):
super().__init__()
self.actor = nn.Sequential(nn.Linear(state_dim, 64), nn.ReLU(), nn.Linear(64, action_dim))
self.critic = nn.Sequential(nn.Linear(state_dim, 64), nn.ReLU(), nn.Linear(64, 1))
def get_action_probs(self, state):
return torch.softmax(self.actor(state), dim=-1)
def get_value(self, state):
return self.critic(state)
model = ActorCritic(state_dim=4, action_dim=2)
# Simulate batch
states = torch.randn(32, 4)
actions = torch.randint(0, 2, (32,))
returns = torch.randn(32) + 2 # Positive returns
with torch.no_grad():
probs = model.get_action_probs(states)
values = model.get_value(states).squeeze()
advantages = returns - values
print(f"Mean advantage: {advantages.mean():.4f}")
assert advantages.shape == (32,), "Should match batch size"
assert torch.isfinite(advantages).all(), "Advantages should be finite"
print("✓ Advantage baseline working")
if __name__ == "__main__":
print("Lab 2: Advantage - PASSED")### Lab 3: Entropy Regularization
import torch
import torch.nn as nn
import numpy as np
class PolicyNet(nn.Module):
def __init__(self, state_dim=4, action_dim=2):
super().__init__()
self.net = nn.Sequential(nn.Linear(state_dim, 64), nn.ReLU(), nn.Linear(64, action_dim))
def forward(self, state):
return torch.softmax(self.net(state), dim=-1)
policy = PolicyNet()
# Compute entropy for batch
states = torch.randn(32, 4)
probs = policy(states)
entropy = -torch.sum(probs * torch.log(probs + 1e-8), dim=1)
mean_entropy = entropy.mean()
print(f"Mean entropy: {mean_entropy:.4f}")
assert entropy.shape == (32,), "Should have entropy per state"
assert (entropy >= 0).all(), "Entropy should be non-negative"
assert mean_entropy < np.log(2), "Max entropy for 2 actions is log(2)"
print("✓ Entropy regularization working")
if __name__ == "__main__":
print("Lab 3: Entropy - PASSED")### Lab 4: Generalized Advantage Estimation
import numpy as np
def compute_gae(rewards, values, gamma=0.99, lambda_=0.95):
advantages = np.zeros_like(rewards)
gae = 0
for t in reversed(range(len(rewards))):
delta = rewards[t] + gamma * values[t+1] - values[t]
gae = delta + gamma * lambda_ * gae
advantages[t] = gae
return advantages
# Simulate trajectory
rewards = np.array([1.0, 0.0, 0.0, 1.0, 1.0])
values = np.array([0.5, 0.3, 0.1, 0.8, 0.9, 0.0])
advantages = compute_gae(rewards, values)
print(f"Advantages: {advantages}")
assert advantages.shape == rewards.shape, "Should match reward shape"
assert np.isfinite(advantages).all(), "Should be finite"
assert advantages[3] > advantages[2], "Later rewards should have higher advantage"
print("✓ GAE working")
if __name__ == "__main__":
print("Lab 4: GAE - PASSED")