Policy Gradient

Policy gradient is a family of reinforcement learning methods that improve a policy directly by nudging its parameters in the direction that makes higher-reward behavior more probable, rather than first learning a table or network of action values and then deriving behavior from it. The policy is a differentiable function, usually a neural network, that maps a state to a probability distribution over actions; training samples actions from it, observes the rewards that follow, and uses gradient ascent on the expected return. This direct approach is what lets the same machinery handle continuous actions, stochastic behavior, and — in modern language models — the choice of the next token among tens of thousands of candidates, where taking an argmax over learned action values would be awkward or impossible.

The founding result is the policy gradient theorem, which says the gradient of expected return can be written as an expectation, over trajectories the policy itself generates, of the gradient of the log-probability of each action multiplied by how good the outcome was. Ronald Williams introduced the simplest estimator in 1992 as REINFORCE, in "Simple statistical gradient-following algorithms for connectionist reinforcement learning": run the policy, record the return that followed each action, and increase the log-probability of actions in proportion to that return. Sutton, McAllester, Singh and Mansour proved the general theorem in 2000, showing that the gradient can be estimated without differentiating through the environment's dynamics at all. The environment can be a black box; only the policy needs to be differentiable, which is why the method applies to games, robots, and text generation alike.

The central practical difficulty is variance: a raw return is a noisy signal, so plain REINFORCE needs enormous numbers of samples and can wander or collapse. The standard remedy is a baseline, a quantity subtracted from the return that does not bias the gradient but removes most of the noise. When the baseline is a learned state-value function, the method becomes an actor-critic: the actor is the policy, the critic estimates value, and the difference between actual and expected return is called the advantage. Generalized advantage estimation, introduced by Schulman and colleagues in 2015, trades bias against variance in that estimate and became a default ingredient of later algorithms.

Proximal Policy Optimization, published by John Schulman and colleagues at OpenAI in 2017, is the variant that became the workhorse, because it makes policy gradients stable enough to use without delicate tuning. An unconstrained step can move the policy so far that performance falls off a cliff and never recovers. Trust Region Policy Optimization (Schulman et al., 2015) addressed this with a constraint on how much the policy may change per update, but it was complicated to implement. PPO keeps the spirit with a simple clipped objective: the probability ratio between the new and old policy is clipped, so an update gains nothing from moving too far. That simplicity let it scale across many parallel workers and made it the optimizer of choice for a decade of reinforcement learning results.

Policy gradients are the engine of reinforcement learning from human feedback (RLHF), where the policy is the language model, an action is a token, and the reward comes from a learned reward model. In the InstructGPT recipe (Ouyang et al., 2022), PPO fine-tunes a supervised model to produce responses the reward model scores highly, while a penalty on KL divergence from the original model keeps the text from drifting into nonsense. The approach has costs: a typical PPO setup holds a policy, a reference model, a reward model and a value model in memory at once. That burden motivated simpler alternatives such as direct preference optimization, which skips the sampling loop, and critic-free variants such as GRPO, used in DeepSeek's 2024 work on mathematical reasoning, which estimate the baseline from a group of sampled answers instead of a separate value network.

A common misconception is that policy gradient methods learn the best action; they in fact learn a distribution, and they are only guaranteed to climb toward a local optimum. They are on-policy by nature, so data gathered by an old policy goes stale quickly, which makes them less sample-efficient than value-based methods such as Q-learning that can reuse old experience. They also optimize whatever reward they are given with great energy, so a flawed reward function or reward model gets exploited rather than corrected, a failure called reward hacking. Entropy bonuses help keep exploration alive, but they do not fix a misspecified objective.

Read policy gradient as learning by doing the thing and reinforcing what worked: sample behavior from a parameterized policy, judge it against a baseline, and shift probability toward the actions that beat expectations — with PPO's clipping as the safety rail that turned this old idea into the standard way to tune a language model against a reward.