reinforcement learning policy gradient

**Policy Gradient and Actor-Critic Methods** are **reinforcement learning algorithms that directly optimize the policy function** — learning to select actions by computing gradients of expected cumulative reward with respect to policy parameters. **Policy Gradient Theorem** - Policy $\pi_\theta(a|s)$: Probability of action $a$ in state $s$, parameterized by $\theta$. - Objective: Maximize $J(\theta) = E_{\tau \sim \pi_\theta}[\sum_t r_t]$. - Gradient: $\nabla_\theta J(\theta) = E[\nabla_\theta \log \pi_\theta(a|s) \cdot Q(s,a)]$ - Key insight: Weight log-probability of actions by their value — increase probability of good actions. **REINFORCE (Williams, 1992)** - Monte Carlo estimate: $g = \sum_t \nabla \log \pi_\theta(a_t|s_t) \cdot G_t$ - $G_t$: Return from step t onward. - High variance — slow to converge. **Actor-Critic** - **Actor**: Policy network $\pi_\theta$ — selects actions. - **Critic**: Value network $V_\phi$ — estimates $V(s)$ to reduce variance. - Advantage: $A(s,a) = Q(s,a) - V(s)$ — how much better action $a$ is vs. average. - Lower variance than REINFORCE; less biased than pure value methods. **A3C (Asynchronous Advantage Actor-Critic)** - Multiple parallel workers independently explore + compute gradients. - Asynchronous updates to shared global network. - Better sample diversity than single-agent training. **PPO (Proximal Policy Optimization)** - Clip objective: $L^{CLIP} = E[\min(r_t A_t, clip(r_t, 1-\epsilon, 1+\epsilon) A_t)]$ - $r_t = \pi_\theta / \pi_{old}$: Probability ratio. - Prevents too-large policy updates — stable training. - Default RL algorithm for RLHF, robotics, game playing. **Reward Shaping** - Sparse rewards: Difficult to learn (reward only at goal). - Reward shaping: Add auxiliary rewards for intermediate progress. - Potential-based shaping: $F(s,a,s') = \gamma\Phi(s') - \Phi(s)$ — provably doesn't change optimal policy. Policy gradient methods are **the core of modern RL** — PPO specifically powers RLHF for LLM alignment and robotic manipulation, making it one of the most practically important algorithms in current AI research.