ddpg

**DDPG** (Deep Deterministic Policy Gradient) is an **off-policy actor-critic algorithm for continuous action spaces** — extending DQN's ideas (replay buffer, target network) to continuous control by learning a deterministic policy that directly outputs continuous actions. **DDPG Components** - **Actor**: Deterministic policy $mu_ heta(s)$ — outputs a continuous action. - **Critic**: Q-network $Q_phi(s, a)$ — estimates the value of state-action pairs. - **Replay Buffer**: Store and replay transitions for off-policy learning — sample efficiency. - **Target Networks**: Soft-updated copies — $ heta' leftarrow au heta + (1- au) heta'$ for stable targets. **Why It Matters** - **Continuous Actions**: DQN can't handle continuous actions (can't enumerate them) — DDPG solves this. - **Off-Policy**: Replay buffer enables sample-efficient, off-policy learning in continuous spaces. - **Foundation**: DDPG is the foundation for TD3 and SAC — the family of continuous control algorithms. **DDPG** is **DQN for continuous actions** — combining a deterministic policy with Q-learning for off-policy continuous control.