policy gradient actor-critic methods
# Policy Gradient & Actor-Critic Methods
## Introduction & Motivation
Policy Gradient: directly optimize policy. Gradient ascent on expected return. Actor-critic: value function baseline. Applications: continuous control, game playing.
Motivation: On-policy learning; continuous actions.
Applications: Control, navigation, manipulation.
---
## Core Concepts & Theory
### Policy Parameterization
Neural network policy; differentiable.
### Advantage Function
Reduce variance; policy gradient.
### Actor-Critic
Separate policy (actor) and value (critic).
---
## Mathematical Formulation
Policy gradient:
$$
abla J( heta) = \mathbb{E}[
abla \log \pi(a|s; heta) A(s,a)]$$
Actor-Critic update:
$$ heta_{\pi} \leftarrow heta_{\pi} + \alpha_{\pi}
abla \log \pi(a|s) A_t$$
$$ heta_v \leftarrow heta_v + \alpha_v (r + \gamma V(s') - V(s))^2$$
---
## Advanced Theory & Extensions
### Proximal Policy Optimization (PPO)
Clipped surrogate objective; stability.
### Advantage Actor-Critic (A2C/A3C)
Parallel workers; scalability.
### Trust Region Policy Optimization
Natural gradient; KL constraint.
---
## Computational Considerations
Policy forward: O(state_dim).
Gradient: O(horizon).
A2C: O(parallel_workers·horizon).
---
## Practical Implementation Strategies
### Baseline Subtraction
Reduce variance with value estimate.
### Generalized Advantage Estimation (GAE)
Trade bias-variance; lambda parameter.
### Entropy Regularization
Encourage exploration.
---
## Benchmark Environments
Continuous Control: MuJoCo tasks.
Game Playing: Atari games.
Robotics: Simulation control.
---
## Key Challenges & Limitations
### High Variance
Policy gradient estimation.
### Sample Efficiency
On-policy; needs many samples.
### Hyperparameter Sensitivity
Step size, baseline weight.
---
## Hyperparameter Tuning
Learning rate: 1e-4 to 3e-4.
Entropy coefficient: 0-0.01.
GAE lambda: 0.95-0.99.
---
## Real-World Applications & Case Studies
Game AI: Continuous strategy games.
Robotics: Joint control, locomotion.
Autonomous Systems: Navigation, manipulation.
---
## Summary & Key Takeaways
Policy Gradient via actor-critic enables on-policy learning through direct policy optimization with value function baseline.
Principles:
1. Policy: parameterized distribution.
2. Policy gradient: expected reward direction.
3. Actor-critic: policy + value.
4. Advantage: variance reduction.
5. GAE: bias-variance trade-off.
---
---
## Appendix: Practical Labs
### Lab 1: Policy Gradient
import numpy as np
def compute_policy_gradient(log_probs, advantages):
"""Compute policy gradient"""
# Policy gradient: E[∇log π(a|s) · A(s,a)]
grad = log_probs * advantages
return grad.mean()
# Test
np.random.seed(42)
log_probs = np.random.randn(64)
advantages = np.random.randn(64)
grad = compute_policy_gradient(log_probs, advantages)
assert np.isfinite(grad), "Gradient finite"
print("✓ Policy gradient working")
if __name__ == "__main__":
print("Lab 1: PolicyGradient - PASSED")### Lab 2: Advantage Estimation
import numpy as np
def compute_gae(rewards, values, gamma=0.99, lambda_gae=0.95):
"""Generalized Advantage Estimation"""
advantages = np.zeros_like(rewards)
gae = 0
for t in reversed(range(len(rewards))):
if t == len(rewards) - 1:
next_value = 0
else:
next_value = values[t + 1]
td_error = rewards[t] + gamma * next_value - values[t]
gae = td_error + gamma * lambda_gae * gae
advantages[t] = gae
return advantages
# Test
np.random.seed(42)
rewards = np.random.rand(50)
values = np.random.rand(50)
advantages = compute_gae(rewards, values)
assert advantages.shape == rewards.shape, "Advantage shape"
print("✓ GAE working")
if __name__ == "__main__":
print("Lab 2: GAE - PASSED")### Lab 3: Actor-Critic Loss
import numpy as np
def actor_critic_loss(log_probs, advantages, value_pred, value_target, entropy, c_ent=0.01):
"""Actor-critic loss"""
# Actor loss (policy)
actor_loss = -(log_probs * advantages).mean()
# Critic loss (value)
critic_loss = (value_target - value_pred) ** 2
# Entropy bonus
entropy_loss = -c_ent * entropy.mean()
# Total loss
total_loss = actor_loss + critic_loss.mean() + entropy_loss
return total_loss
# Test
np.random.seed(42)
log_p = np.random.randn(32)
adv = np.random.randn(32)
v_pred = np.random.randn(32)
v_targ = np.random.randn(32)
entropy = np.random.rand(32)
loss = actor_critic_loss(log_p, adv, v_pred, v_targ, entropy)
assert np.isfinite(loss), "Loss finite"
print("✓ Actor-Critic loss working")
if __name__ == "__main__":
print("Lab 3: ActorCriticLoss - PASSED")### Lab 4: PPO Clipped Surrogate
import numpy as np
def ppo_clipped_loss(log_probs_new, log_probs_old, advantages, epsilon=0.2):
"""PPO clipped surrogate loss"""
# Probability ratio
ratio = np.exp(log_probs_new - log_probs_old)
# Clipped objective
clipped_ratio = np.clip(ratio, 1 - epsilon, 1 + epsilon)
surrogate = np.minimum(ratio * advantages, clipped_ratio * advantages)
# Loss (negative for gradient ascent)
loss = -surrogate.mean()
return loss
# Test
np.random.seed(42)
log_new = np.random.randn(32)
log_old = np.random.randn(32)
adv = np.random.randn(32)
loss = ppo_clipped_loss(log_new, log_old, adv)
assert np.isfinite(loss), "Loss finite"
print("✓ PPO loss working")
if __name__ == "__main__":
print("Lab 4: PPOLoss - PASSED")