Actor-Critic Methods A3c Sac Hybrid Algorithms
# Actor-Critic Methods: A3C, SAC & Hybrid Algorithms
## Introduction & Motivation
Actor-critic: combine policy (actor) and value (critic) functions. A3C: asynchronous advantage actor-critic; parallel workers. SAC: soft actor-critic; maximum entropy; continuous control. TD advantage: G_t - V(s); reduce variance. Applications: continuous control, game playing, robotics.
Motivation: Policy gradient: high variance. Value: biased but low variance. Combine for best of both.
Applications: Continuous control, robotic manipulation, game AI.
---
## Core Concepts & Theory
### Actor
Policy π(a|s); learns to maximize expected return.
### Critic
Value V(s); learns to estimate return; provides baseline.
### Advantage
A(s,a) = Q(s,a) - V(s); reduces variance.
---
## Mathematical Formulation
Actor update:
$$
abla J( heta) = \mathbb{E}[
abla_ heta \log \pi(a|s) A(s,a)]$$
Critic update:
$$L_V = \mathbb{E}[(V(s) - (r + \gamma V(s')))^2]$$
SAC entropy:
$$L = -\mathbb{E}[log \pi_ heta(a|s) - \alpha \log \pi_ heta(a|s) - Q(s,a)]$$
---
## Advanced Theory & Extensions
### A2C (Advantage Actor-Critic)
Synchronous; single worker version of A3C.
### PPO (Proximal Policy Optimization)
Clip ratio; stable policy updates.
### TRPO (Trust Region)
Natural gradient; theoretical convergence.
---
## Computational Considerations
A3C: O(workers × steps) parallel computation.
SAC: O(env + 2 networks) per step; efficient.
Training: Continuous on-policy updates.
---
## Practical Implementation Strategies
### Shared Backbone
Actor and critic share early layers; reduce parameters.
### Learning Rate
Different LR for actor/critic; typically critic lower.
### Entropy Coefficient
α controls exploration; learned or fixed.
---
## Benchmark Datasets & Evaluation
MuJoCo: Continuous control; average return.
Atari: Discrete; combined with DQN.
Robotic: Sim2Real; continuous control.
---
## Key Challenges & Limitations
### Convergence
No convergence guarantees; sometimes unstable.
### Hyperparameter Tuning
Multiple hyperparameters; sensitive.
### Exploration
Needs careful entropy regularization.
---
## Hyperparameter Tuning
Actor LR: 1e-4 typical.
Critic LR: 3e-4 common; faster convergence.
Entropy α: auto-adjusted; or fixed 0.2.
---
## Real-World Applications & Case Studies
Robotic Manipulation: SAC standard; continuous actions.
Game AI: A3C for Atari; competitive performance.
Autonomous Driving: Continuous steering + acceleration.
---
## Integration with Other Methods
Actor-Critic + Experience Replay → off-policy variants.
Actor-Critic + Prioritized Replay → stable learning.
---
## Summary & Key Takeaways
Actor-critic methods combine policy and value learning, leveraging advantages of both policy gradient and value methods through shared representations and entropy regularization.
Principles:
1. Actor: policy gradient with advantage baseline.
2. Critic: value function estimates return.
3. Advantage: A(s,a) = Q(s,a) - V(s).
4. Entropy: encourages exploration; SAC standard.
5. Parallel: A3C asynchronous workers.
---
---
## Appendix: Practical Labs
### Lab 1: Actor-Critic Training
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
class ActorCriticAgent:
def __init__(self, state_dim, action_dim, hidden_dim=64):
self.actor = nn.Sequential(
nn.Linear(state_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, action_dim),
nn.Softmax(dim=1)
)
self.critic = nn.Sequential(
nn.Linear(state_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, 1)
)
def get_value(self, state):
return self.critic(state)
def get_action_prob(self, state):
return self.actor(state)
# Test
np.random.seed(42)
agent = ActorCriticAgent(state_dim=10, action_dim=4)
state = torch.randn(8, 10)
value = agent.get_value(state)
probs = agent.get_action_prob(state)
assert value.shape == (8, 1), "Value shape"
assert probs.shape == (8, 4), "Probs shape"
print("✓ Actor-critic agent working")
if __name__ == "__main__":
print("Lab 1: Agent - PASSED")### Lab 2: TD Advantage Estimation
import numpy as np
def compute_td_advantage(rewards, values, gamma=0.99):
"""Compute TD advantage: r + γV(s') - V(s)"""
advantages = []
for t in range(len(rewards)):
if t < len(rewards) - 1:
td_error = rewards[t] + gamma * values[t+1] - values[t]
else:
td_error = rewards[t] - values[t]
advantages.append(td_error)
return np.array(advantages)
# Test
np.random.seed(42)
rewards = [1.0, 2.0, 1.5, 0.5]
values = [0.5, 1.0, 1.2, 0.8]
advantages = compute_td_advantage(rewards, values)
assert len(advantages) == 4, "Advantages length"
assert np.isfinite(advantages).all(), "All finite"
print("✓ TD advantage working")
if __name__ == "__main__":
print("Lab 2: Advantage - PASSED")### Lab 3: Entropy Regularization
import torch
import torch.nn.functional as F
import numpy as np
def compute_entropy(probs):
"""Compute entropy of probability distribution"""
entropy = -torch.sum(probs * torch.log(probs + 1e-8), dim=1)
return entropy
def sac_loss(actor, critic, state, action, reward, next_state, alpha=0.2):
"""SAC loss: maximize return + entropy"""
value = critic(state)
next_value = critic(next_state)
td_target = reward + 0.99 * next_value
critic_loss = F.mse_loss(value, td_target.detach())
probs = actor(state)
entropy = compute_entropy(probs)
# Actor loss: maximize log prob + entropy
log_probs = torch.log(probs + 1e-8)
actor_loss = -(log_probs.sum(dim=1) + alpha * entropy).mean()
return actor_loss, critic_loss
# Test
np.random.seed(42)
actor = nn.Linear(10, 4)
critic = nn.Linear(10, 1)
state = torch.randn(8, 10)
action = torch.randint(0, 4, (8,))
reward = torch.rand(8)
next_state = torch.randn(8, 10)
a_loss, c_loss = sac_loss(actor, critic, state, action, reward, next_state)
assert torch.isfinite(a_loss), "Actor loss finite"
assert torch.isfinite(c_loss), "Critic loss finite"
print("✓ SAC loss working")
if __name__ == "__main__":
print("Lab 3: SAC - PASSED")### Lab 4: Parallel A3C Simulation
import numpy as np
def a3c_worker_step(state, reward, done, gamma=0.99):
"""Simulate single A3C worker step"""
# Accumulate gradients
grad_actor = np.random.randn(10) * 0.01
grad_critic = np.random.randn(1) * 0.01
# TD error
td_error = reward - state.sum() # Simple heuristic
return grad_actor, grad_critic, td_error
# Test
np.random.seed(42)
state = np.random.randn(10)
reward = 1.0
grad_a, grad_c, td_error = a3c_worker_step(state, reward, done=False)
assert grad_a.shape == (10,), "Actor gradient shape"
assert grad_c.shape == (1,), "Critic gradient shape"
print("✓ A3C worker working")
if __name__ == "__main__":
print("Lab 4: A3C - PASSED")