reinforcement learning policy gradient

**Deep Reinforcement Learning** is **the artificial intelligence paradigm where agents learn optimal behavior through trial-and-error interaction with environments — combining deep neural networks as function approximators with reinforcement learning algorithms to handle high-dimensional state spaces, enabling mastery of games, robotic control, and complex decision-making tasks**. **Value-Based Methods:** - **Q-Learning**: learns action-value function Q(s,a) estimating expected cumulative reward for taking action a in state s — agent selects action with highest Q-value; tabular Q-learning works for small state spaces - **Deep Q-Network (DQN)**: neural network approximates Q-function for high-dimensional states (e.g., raw pixels) — key innovations: experience replay (randomly sample past transitions), target network (slowly updated copy for stable targets), and ε-greedy exploration - **Double DQN**: addresses Q-value overestimation by using online network for action selection and target network for value estimation — reduces positive bias that causes suboptimal policies in standard DQN - **Dueling DQN**: separates Q(s,a) into state value V(s) and advantage A(s,a) streams — V(s) estimates how good a state is regardless of action; A(s,a) estimates relative advantage of each action; improves learning for states where action choice matters less **Policy Gradient Methods:** - **REINFORCE**: directly optimizes policy π(a|s;θ) by gradient ascent on expected reward — ∇J(θ) = E[∇log π(a|s;θ) × R]; high variance requires baseline subtraction (typically V(s)) for practical convergence - **Actor-Critic**: actor (policy network) selects actions, critic (value network) estimates expected return — advantage A(s,a) = Q(s,a) - V(s) reduces variance compared to pure policy gradient; TD error provides online update signal - **PPO (Proximal Policy Optimization)**: clips policy ratio to prevent destructively large updates — L^CLIP = min(r_t(θ)A_t, clip(r_t(θ), 1-ε, 1+ε)A_t) where r_t is probability ratio of new/old policy; stable training without careful learning rate tuning - **SAC (Soft Actor-Critic)**: maximizes reward plus entropy bonus — encourages exploration by penalizing deterministic policies; achieves robust performance across continuous control tasks; automatic temperature adjustment **Exploration vs. Exploitation:** - **ε-Greedy**: with probability ε take random action, otherwise take greedy action — simple but uniform random exploration is inefficient in large action spaces - **Intrinsic Motivation**: reward agent for visiting novel states — curiosity-driven exploration using prediction error of learned world model; count-based exploration bonuses for rarely visited states - **Reward Shaping**: engineer intermediate rewards to guide learning toward distant goals — must preserve optimal policy (potential-based shaping); helps bridge sparse reward signals in long-horizon tasks **Deep reinforcement learning has achieved superhuman performance in Atari games (DQN), Go (AlphaGo/AlphaZero), StarCraft II (AlphaStar), and robotic manipulation — representing the frontier of AI systems that learn complex behaviors through environmental interaction rather than supervised data.**