Td3 - Twin Delayed Ddpg
# TD3 - Twin Delayed DDPG
## Introduction & Motivation
TD3: address DDPG overestimation through twin Q-networks and delayed updates. Improves stability and convergence reliability. Double estimation reduces bias accumulation in continuous control.
Motivation: Reduce overestimation bias for more stable continuous control.
Applications: Robotics, complex continuous tasks, high-dimensional control.
---
## Core Concepts & Theory
### Twin Q-Networks
Use two Q-networks for value estimation.
### Delayed Policy Update
Update actor less frequently than critic.
### Target Policy Smoothing
Add noise to target actions.
### Clipped Double Q-Learning
Take minimum of two Q-estimates.
---
## Mathematical Formulation
TD3 Update:
$$Q(s,a) \leftarrow \min(Q_1, Q_2) + r + \gamma \min(Q'_1, Q'_2)$$
Delayed Actor:
$$ heta \leftarrow heta + \alpha
abla_ heta Q_1(s, \mu_ heta(s))$$
Target Policy Smoothing:
$$a' = \mu'(s') + ext{clip}(\mathcal{N}(0, \sigma), -c, c)$$
---
## Advanced Theory & Extensions
### Clipped Double Backup
Prevent overestimation.
### Policy Smoothing
Reduce target error variance.
### Update Frequency
Asymmetric critic-actor updates.
---
## Computational Considerations
Dual Q-Networks: 2x critic computation.
Delayed Updates: 1/d actor updates.
Total: ~2.3x DDPG cost (d=2).
---
## Practical Implementation Strategies
### Network Initialization
Separate Q-network initialization.
### Noise Schedules
Decay exploration and smoothing.
### Update Coordination
Synchronize network copies.
---
## Benchmark Datasets & Evaluation
MuJoCo: Continuous control benchmark.
Robotics: Manipulation and locomotion.
Benchmarks: Standard RL evaluation.
---
## Key Challenges & Limitations
### Double Computation
Higher memory and compute cost.
### Hyperparameter Tuning
Policy update frequency critical.
### Convergence Speed
Delayed updates may slow learning.
---
## Hyperparameter Tuning
Policy update frequency: Every 2 steps.
Target smoothing σ: 0.2.
Clip range c: 0.5.
---
## Real-World Applications & Case Studies
Robot Learning: Dexterous manipulation.
Autonomous Systems: Continuous navigation.
Complex Control: High-dimensional tasks.
---
## Integration with Other Methods
TD3 + prioritized replay; + hindsight experience replay.
---
## Summary & Key Takeaways
TD3 improves DDPG through twin networks and delayed updates.
Principles:
1. Twin: Dual Q-network estimation.
2. Clipping: Minimum for bias reduction.
3. Delayed: Infrequent actor updates.
4. Smoothing: Target noise injection.
5. Stability: Reduced overestimation.
---
## Appendix: Practical Labs
### Lab 1: Twin Q-Networks
import numpy as np
def twin_q_update(Q1, Q2, state, action, reward, next_state,
actor_target, Q1_target, Q2_target, gamma=0.99):
"""TD3 twin Q-learning update"""
# Target policy with smoothing
next_action = actor_target(next_state) + np.random.randn(2) * 0.2
next_action = np.clip(next_action, -1, 1)
# Clipped double backup
q1_next = Q1_target(next_state, next_action)
q2_next = Q2_target(next_state, next_action)
target_q = reward + gamma * np.minimum(q1_next, q2_next)
# Update both Q-networks
current_q1 = Q1(state, action)
current_q2 = Q2(state, action)
loss1 = (current_q1 - target_q) ** 2
loss2 = (current_q2 - target_q) ** 2
return loss1, loss2
state = np.random.randn(10)
action = np.random.randn(2)
reward = 1.0
next_state = np.random.randn(10)
Q1 = lambda s, a: np.random.randn()
Q2 = lambda s, a: np.random.randn()
actor_target = lambda s: np.random.randn(2)
Q1_target = lambda s, a: np.random.randn()
Q2_target = lambda s, a: np.random.randn()
l1, l2 = twin_q_update(Q1, Q2, state, action, reward, next_state, actor_target, Q1_target, Q2_target)
print(f"✓ Twin Q-update: loss1={l1:.3f}, loss2={l2:.3f}")### Lab 2: Delayed Policy Update
import numpy as np
def delayed_actor_update(actor, Q1, state, learning_rate=1e-3, update_freq=2, step=0):
"""Update actor every d steps"""
if step % update_freq == 0:
# Policy gradient
action = actor(state)
q_value = Q1(state, action)
gradient = -q_value # Maximize Q
# Actor update
actor_weights = actor.weights + learning_rate * gradient
return actor_weights
else:
return actor.weights
class SimpleActor:
def __init__(self):
self.weights = np.random.randn(10, 2) * 0.01
def __call__(self, state):
return state @ self.weights
actor = SimpleActor()
Q1 = lambda s, a: (s @ a).sum()
for step in range(4):
w = delayed_actor_update(actor, Q1, np.random.randn(10), update_freq=2, step=step)
print(f"✓ Step {step}: {'Updated' if step % 2 == 0 else 'Skipped'}")### Lab 3: Target Policy Smoothing
import numpy as np
def target_policy_with_smoothing(policy, state, noise_scale=0.2, clip_range=0.5):
"""Apply smoothing to target policy"""
action = policy(state)
# Add noise
noise = np.random.randn(*action.shape) * noise_scale
smoothed_action = action + noise
# Clip to valid range
smoothed_action = np.clip(smoothed_action, -clip_range, clip_range)
return smoothed_action
policy = lambda s: (s @ np.random.randn(10, 2)).mean(axis=-1, keepdims=True)
state = np.random.randn(10)
action_smooth = target_policy_with_smoothing(policy, state, noise_scale=0.2)
print(f"✓ Smoothed action: {action_smooth}")### Lab 4: Complete TD3 Agent
import numpy as np
class TD3Agent:
def __init__(self, state_dim, action_dim):
self.state_dim = state_dim
self.action_dim = action_dim
# Twin critics
self.Q1 = np.random.randn(state_dim + action_dim, 1) * 0.01
self.Q2 = np.random.randn(state_dim + action_dim, 1) * 0.01
# Target critics
self.Q1_target = self.Q1.copy()
self.Q2_target = self.Q2.copy()
# Actor
self.actor = np.random.randn(state_dim, action_dim) * 0.01
self.actor_target = self.actor.copy()
self.step = 0
def select_action(self, state):
"""Select action with noise"""
action = state @ self.actor
action += np.random.randn(self.action_dim) * 0.1
return np.clip(action, -1, 1)
def update(self, state, action, reward, next_state, done):
"""TD3 update"""
self.step += 1
# Compute target
next_action = (next_state @ self.actor_target) + np.random.randn(self.action_dim) * 0.2
next_action = np.clip(next_action, -1, 1)
q1_next = (np.concatenate([next_state, next_action]) @ self.Q1_target)[0]
q2_next = (np.concatenate([next_state, next_action]) @ self.Q2_target)[0]
target = reward if done else reward + 0.99 * min(q1_next, q2_next)
# Critic loss
sa = np.concatenate([state, action])
loss = (sa @ self.Q1 - target) ** 2 + (sa @ self.Q2 - target) ** 2
# Delayed actor update
if self.step % 2 == 0:
actor_loss = -(sa @ self.Q1)
else:
actor_loss = 0
return loss
agent = TD3Agent(state_dim=10, action_dim=2)
state = np.random.randn(10)
action = agent.select_action(state)
loss = agent.update(state, action, 1.0, state*0.9, False)
print(f"✓ TD3 agent: action={action[:2]}, loss={loss:.3f}")---