Sac - Soft Actor-Critic
# SAC - Soft Actor-Critic
## Introduction & Motivation
SAC: maximum entropy reinforcement learning with entropy regularization. Automatic temperature tuning for exploration-exploitation balance. Sample-efficient continuous control with stochastic policies.
Motivation: Learn stochastic policies balancing reward and entropy.
Applications: Robotics, continuous control, exploration-critical tasks.
---
## Core Concepts & Theory
### Entropy Regularization
Encourage exploration through policy entropy.
### Automatic Temperature Tuning
Self-adjusting entropy coefficient.
### Stochastic Policy
Output probability distribution over actions.
### Twin Q-Networks
Reduce overestimation bias.
---
## Mathematical Formulation
Maximum Entropy Objective:
$$J(\pi) = \mathbb{E}[Q(s,a) - \alpha \log \pi(a|s)]$$
Automatic Temperature:
$$\alpha \leftarrow \alpha - \beta
abla_\alpha \mathbb{E}[-\alpha \log \pi(a|s) - \alpha \mathcal{H}]$$
Q-Target:
$$Q(s,a) \leftarrow r + \gamma (Q'(s',a') - \alpha \log \pi(a'|s'))$$
---
## Advanced Theory & Extensions
### Temperature Scheduling
Decay entropy coefficient.
### Reparameterization Trick
Differentiable sampling for gradients.
### Multi-Agent SAC
Centralized training, decentralized execution.
---
## Computational Considerations
Entropy Computation: O(D·A).
Temperature Update: O(1).
Total: O(D·(A+1)) per sample.
---
## Practical Implementation Strategies
### Reparameterization
μ(s) and σ(s) parameterization.
### Entropy Target
Set minimum target entropy.
### Learning Rates
Separate for actor, critic, alpha.
---
## Benchmark Datasets & Evaluation
MuJoCo: Continuous control.
Complex Manipulation: Dexterous tasks.
Sample Efficiency: Data efficiency metrics.
---
## Key Challenges & Limitations
### Temperature Tuning
Sensitive to entropy target.
### Convergence
Complex objective with multiple terms.
### Computational Overhead
Multiple Q-networks and critic.
---
## Hyperparameter Tuning
Target entropy: -A (action dimension).
Temperature learning rate: 1e-4.
Actor learning rate: 3e-4.
---
## Real-World Applications & Case Studies
Robot Manipulation: Safe exploration.
Autonomous Driving: Robust policies.
Complex Environments: Exploration-heavy tasks.
---
## Integration with Other Methods
SAC + hindsight experience replay; + domain randomization.
---
## Summary & Key Takeaways
SAC learns maximum entropy policies for robust control.
Principles:
1. Entropy: Maximize exploration.
2. Stochastic: Probabilistic actions.
3. Temperature: Automatic tuning.
4. Twin Q: Reduce bias.
5. Reparameterization: Differentiable sampling.
---
## Appendix: Practical Labs
### Lab 1: Reparameterization Trick
import numpy as np
def reparameterized_sample(mu, log_sigma, num_samples=1):
"""Sample using reparameterization trick"""
epsilon = np.random.randn(num_samples, len(mu))
sigma = np.exp(log_sigma)
samples = mu + sigma * epsilon
return samples
def log_probability(action, mu, log_sigma):
"""Compute log probability under Gaussian"""
sigma = np.exp(log_sigma)
log_prob = -0.5 * np.sum(((action - mu) / sigma) ** 2) - np.sum(log_sigma)
return log_prob
mu = np.array([0.5, -0.3])
log_sigma = np.array([-0.5, -0.7])
samples = reparameterized_sample(mu, log_sigma, num_samples=10)
log_prob = log_probability(samples[0], mu, log_sigma)
print(f"✓ Reparameterized samples: shape={samples.shape}")
print(f"✓ Log probability: {log_prob:.3f}")### Lab 2: Entropy Regularization
import numpy as np
def compute_entropy(log_probs):
"""Compute policy entropy"""
entropy = -np.mean(log_probs)
return entropy
def entropy_regularized_loss(q_value, log_prob, alpha=0.2):
"""Compute SAC objective"""
# Maximize Q - alpha * log_prob
loss = -(q_value - alpha * log_prob)
return loss
log_probs = np.array([-1.0, -1.5, -0.8, -2.0])
q_values = np.array([1.0, 0.5, 1.2, 0.3])
entropy = compute_entropy(log_probs)
sac_loss = entropy_regularized_loss(q_values, log_probs, alpha=0.2)
print(f"✓ Entropy: {entropy:.3f}")
print(f"✓ SAC loss: {sac_loss:.3f}")### Lab 3: Automatic Temperature Update
import numpy as np
def update_temperature(log_alpha, log_probs, target_entropy, lr=1e-4):
"""Update entropy coefficient automatically"""
alpha = np.exp(log_alpha)
# Temperature gradient
entropy_error = -(np.mean(log_probs) - target_entropy)
log_alpha_update = lr * entropy_error
# Update log_alpha
log_alpha_new = log_alpha + log_alpha_update
alpha_new = np.exp(log_alpha_new)
return log_alpha_new, alpha_new
log_alpha = np.log(0.2)
log_probs = np.random.randn(32) - 1.0
target_entropy = -2 # Negative for continuous actions
log_alpha_new, alpha_new = update_temperature(log_alpha, log_probs, target_entropy)
print(f"✓ Temperature: α={np.exp(log_alpha):.3f} -> {alpha_new:.3f}")### Lab 4: Complete SAC Agent
import numpy as np
class SACAgent:
def __init__(self, state_dim, action_dim, target_entropy=None):
self.state_dim = state_dim
self.action_dim = action_dim
# Networks
self.actor_mu = np.random.randn(state_dim, action_dim) * 0.01
self.actor_log_sigma = np.random.randn(state_dim, action_dim) * 0.01
self.Q1 = np.random.randn(state_dim + action_dim, 1) * 0.01
self.Q2 = np.random.randn(state_dim + action_dim, 1) * 0.01
self.Q1_target = self.Q1.copy()
self.Q2_target = self.Q2.copy()
# Temperature
self.log_alpha = np.log(0.2)
self.target_entropy = target_entropy or -action_dim
def select_action(self, state):
"""Select action from stochastic policy"""
mu = state @ self.actor_mu
log_sigma = state @ self.actor_log_sigma
epsilon = np.random.randn(self.action_dim)
sigma = np.exp(log_sigma)
action = mu + sigma * epsilon
return np.tanh(action)
def compute_log_prob(self, state, action):
"""Compute log probability"""
mu = state @ self.actor_mu
log_sigma = state @ self.actor_log_sigma
log_prob = -0.5 * ((action - mu) ** 2) / np.exp(2 * log_sigma)
log_prob = log_prob.sum() - np.sum(log_sigma)
return log_prob
def update(self, state, action, reward, next_state, done, gamma=0.99):
"""SAC update"""
alpha = np.exp(self.log_alpha)
# Critic update
next_action = self.select_action(next_state)
next_log_prob = self.compute_log_prob(next_state, next_action)
sa = np.concatenate([next_state, next_action])
q1_next = (sa @ self.Q1_target)[0]
q2_next = (sa @ self.Q2_target)[0]
target = reward if done else reward + gamma * (min(q1_next, q2_next) - alpha * next_log_prob)
# Actor update
log_prob = self.compute_log_prob(state, action)
actor_loss = -(min(q1_next, q2_next) - alpha * log_prob)
# Temperature update
alpha_loss = -alpha * (log_prob + self.target_entropy)
return actor_loss, alpha_loss
agent = SACAgent(state_dim=10, action_dim=2)
state = np.random.randn(10)
action = agent.select_action(state)
a_loss, α_loss = agent.update(state, action, 1.0, state*0.9, False)
print(f"✓ SAC agent: action shape={action.shape}, actor_loss={a_loss:.3f}")---