td3

**TD3** (Twin Delayed DDPG) is an **improvement to DDPG for continuous control that addresses overestimation bias** — using twin critics, delayed policy updates, and target policy smoothing for stable, high-performance actor-critic learning. **TD3 Innovations** - **Twin Critics**: Two Q-networks — use the minimum $min(Q_1, Q_2)$ to compute targets, reducing overestimation. - **Delayed Policy Updates**: Update the actor less frequently than the critics (every 2 critic updates) — more stable learning. - **Target Smoothing**: Add noise to the target action: $a' = pi_{target}(s') + ext{clip}(N(0, sigma), -c, c)$ — regularizes the value function. - **Deterministic Policy**: The actor outputs deterministic actions — exploration via added Gaussian noise. **Why It Matters** - **Overestimation Fix**: Twin critics + target smoothing dramatically reduce Q-value overestimation. - **Stability**: Delayed updates prevent the policy from exploiting estimation errors in the Q-function. - **Performance**: TD3 significantly outperforms DDPG on MuJoCo continuous control benchmarks. **TD3** is **DDPG done right** — fixing overestimation and instability with twin critics, delayed updates, and target smoothing.