trust region policy optimization

**TRPO** (Trust Region Policy Optimization) is a **policy gradient RL algorithm that constrains policy updates to a trust region** — ensuring that each update doesn't change the policy too much, providing theoretical monotonic improvement guarantees. **TRPO Algorithm** - **Constraint**: Limit the KL divergence between old and new policies: $D_{KL}(pi_{old} | pi_{new}) leq delta$. - **Optimization**: $max_ heta mathbb{E}[frac{pi_ heta(a|s)}{pi_{old}(a|s)} A(s,a)]$ subject to the KL constraint. - **Solving**: Uses conjugate gradient + line search to approximately solve the constrained optimization. - **Natural Gradient**: TRPO is equivalent to a natural gradient step — accounts for the policy's geometry. **Why It Matters** - **Monotonic Improvement**: Each TRPO update is guaranteed to improve (or not decrease) the expected return. - **Stability**: KL constraint prevents destructive large policy updates — stable training. - **Foundation**: TRPO laid the theoretical foundation for PPO — PPO simplifies TRPO's constrained optimization. **TRPO** is **safe policy updates** — constraining each step to a trust region for guaranteed monotonic improvement in reinforcement learning.