proximal policy optimization

**PPO** (Proximal Policy Optimization) is the **most widely used policy gradient RL algorithm** — simplifying TRPO's constrained optimization into a clipped surrogate objective that achieves similar stability with much simpler implementation and better empirical performance. **PPO Clipped Objective** - **Ratio**: $r_t( heta) = frac{pi_ heta(a_t|s_t)}{pi_{old}(a_t|s_t)}$ — probability ratio between new and old policy. - **Clipped**: $L^{CLIP} = min(r_t A_t, ext{clip}(r_t, 1-epsilon, 1+epsilon) A_t)$ — clip the ratio to $[1-epsilon, 1+epsilon]$. - **$epsilon$ Parameter**: Typically 0.1-0.2 — controls how much the policy can change per update. - **Mini-Batch**: Multiple optimization epochs per data collection — more sample efficient than vanilla policy gradient. **Why It Matters** - **Simplicity**: Much simpler than TRPO — no conjugate gradient, no KL constraint, just clipping. - **RLHF**: PPO is the standard algorithm for RLHF (Reinforcement Learning from Human Feedback) in LLMs. - **Versatility**: Works for discrete and continuous actions, single and multi-agent, games and robotics. **PPO** is **the workhorse of modern RL** — simple, stable, and effective policy optimization through clipped surrogate objectives.