reinforcement learning deep

**Deep Reinforcement Learning (Deep RL)** is the **machine learning paradigm where neural networks learn optimal sequential decision-making policies through trial-and-error interaction with an environment — receiving reward signals that guide the agent toward maximizing cumulative long-term returns, enabling superhuman performance on video games, robotic control, chip placement, and serving as the foundation for RLHF in language model alignment**. **Core Framework** An agent observes state s, takes action a according to policy π(a|s), receives reward r, and transitions to next state s'. The goal is to learn π that maximizes the expected cumulative discounted reward: E[Σ γᵗ rₜ], where γ ∈ [0,1) is the discount factor. **Value-Based Methods** - **DQN (Deep Q-Network)**: Learn Q(s,a) — the expected return of taking action a in state s and following the optimal policy thereafter. The policy is implicitly: take the action with highest Q-value. Key innovations: experience replay (store and sample past transitions), target network (stable training target updated periodically). First to achieve superhuman Atari play. - **Rainbow DQN**: Combines six DQN improvements: double Q-learning, prioritized replay, dueling architecture, multi-step returns, distributional RL, noisy networks. State-of-the-art value-based performance. **Policy Gradient Methods** - **REINFORCE**: Directly optimize the policy πθ by gradient ascent on expected return: ∇J = E[∇log πθ(a|s) · G], where G is the return. High variance — requires many samples. - **PPO (Proximal Policy Optimization)**: Clips the policy ratio to prevent large updates: L = min(rθ · A, clip(rθ, 1-ε, 1+ε) · A), where rθ = πnew/πold and A is the advantage. Simple, stable, widely used. The RL algorithm used in RLHF (ChatGPT, Claude). - **TRPO (Trust Region Policy Optimization)**: Constrains each policy update to stay within a KL-divergence trust region of the old policy. More theoretically principled than PPO but harder to implement. **Actor-Critic Methods** - **A3C/A2C**: Combine policy (actor) and value function (critic) networks. The critic estimates V(s) to reduce gradient variance; the actor updates using advantage A = r + γV(s') - V(s). - **SAC (Soft Actor-Critic)**: Maximizes both return and policy entropy, encouraging exploration and robustness. State-of-the-art for continuous control (robotics, locomotion). **Challenges** - **Sample Efficiency**: Deep RL typically requires millions of environment interactions. Transfer learning and offline RL (learning from logged data) partially address this. - **Reward Design**: Sparse or misspecified rewards lead to poor learning or reward hacking. Reward shaping, intrinsic motivation (curiosity-driven exploration), and inverse RL help. - **Stability**: The non-stationarity of RL (the data distribution changes as the policy improves) makes training unstable. Replay buffers, target networks, and conservative policy updates mitigate this. Deep Reinforcement Learning is **the framework that teaches neural networks to act, not just perceive** — connecting perception to action through reward-driven optimization and enabling AI systems that learn complex behaviors from experience.