Deep Reinforcement Learning (Deep RL) is the machine learning paradigm where neural networks learn optimal sequential decision-making policies through trial-and-error interaction with an environment — receiving reward signals that guide the agent toward maximizing cumulative long-term returns, enabling superhuman performance on video games, robotic control, chip placement, and serving as the foundation for RLHF in language model alignment.

Core Framework

An agent observes state s, takes action a according to policy π(a|s), receives reward r, and transitions to next state s'. The goal is to learn π that maximizes the expected cumulative discounted reward: E[Σ γᵗ rₜ], where γ ∈ [0,1) is the discount factor.

Value-Based Methods

• DQN (Deep Q-Network): Learn Q(s,a) — the expected return of taking action a in state s and following the optimal policy thereafter. The policy is implicitly: take the action with highest Q-value. Key innovations: experience replay (store and sample past transitions), target network (stable training target updated periodically). First to achieve superhuman Atari play.
• Rainbow DQN: Combines six DQN improvements: double Q-learning, prioritized replay, dueling architecture, multi-step returns, distributional RL, noisy networks. State-of-the-art value-based performance.

Policy Gradient Methods

• REINFORCE: Directly optimize the policy πθ by gradient ascent on expected return: ∇J = E[∇log πθ(a|s) · G], where G is the return. High variance — requires many samples.
• PPO (Proximal Policy Optimization): Clips the policy ratio to prevent large updates: L = min(rθ · A, clip(rθ, 1-ε, 1+ε) · A), where rθ = πnew/πold and A is the advantage. Simple, stable, widely used. The RL algorithm used in RLHF (ChatGPT, Claude).
• TRPO (Trust Region Policy Optimization): Constrains each policy update to stay within a KL-divergence trust region of the old policy. More theoretically principled than PPO but harder to implement.

Actor-Critic Methods

• A3C/A2C: Combine policy (actor) and value function (critic) networks. The critic estimates V(s) to reduce gradient variance; the actor updates using advantage A = r + γV(s') - V(s).
• SAC (Soft Actor-Critic): Maximizes both return and policy entropy, encouraging exploration and robustness. State-of-the-art for continuous control (robotics, locomotion).

Challenges

• Sample Efficiency: Deep RL typically requires millions of environment interactions. Transfer learning and offline RL (learning from logged data) partially address this.
• Reward Design: Sparse or misspecified rewards lead to poor learning or reward hacking. Reward shaping, intrinsic motivation (curiosity-driven exploration), and inverse RL help.
• Stability: The non-stationarity of RL (the data distribution changes as the policy improves) makes training unstable. Replay buffers, target networks, and conservative policy updates mitigate this.

Deep Reinforcement Learning is the framework that teaches neural networks to act, not just perceive — connecting perception to action through reward-driven optimization and enabling AI systems that learn complex behaviors from experience.