DDPG (Deep Deterministic Policy Gradient) is an off-policy actor-critic algorithm for continuous action spaces — extending DQN's ideas (replay buffer, target network) to continuous control by learning a deterministic policy that directly outputs continuous actions.

DDPG Components

• Actor: Deterministic policy $mu_ heta(s)$ — outputs a continuous action.
• Critic: Q-network $Q_phi(s, a)$ — estimates the value of state-action pairs.
• Replay Buffer: Store and replay transitions for off-policy learning — sample efficiency.
• Target Networks: Soft-updated copies — $ heta' leftarrow au heta + (1- au) heta'$ for stable targets.

Why It Matters

• Continuous Actions: DQN can't handle continuous actions (can't enumerate them) — DDPG solves this.
• Off-Policy: Replay buffer enables sample-efficient, off-policy learning in continuous spaces.
• Foundation: DDPG is the foundation for TD3 and SAC — the family of continuous control algorithms.

DDPG is DQN for continuous actions — combining a deterministic policy with Q-learning for off-policy continuous control.