ppo adaptive kl
**PPO with Adaptive KL** is a variant of Proximal Policy Optimization that dynamically adjusts the KL divergence penalty coefficient during training based on observed policy changes.
## What Is Adaptive KL in PPO?
- **Mechanism**: Increases penalty when KL exceeds target, decreases when below
- **Target KL**: Typically 0.01-0.02 for stable training
- **Adaptation Rate**: Usually 1.5× increase or 0.5× decrease per update
- **Alternative**: PPO-Clip uses hard clipping instead of adaptive penalty
## Why Adaptive KL Matters
Fixed KL coefficients either over-constrain learning (too high) or allow destructive updates (too low). Adaptive tuning maintains stable training across different phases.
```python
# Adaptive KL coefficient update
target_kl = 0.01
kl_coef = 0.2 # Initial coefficient
for epoch in training:
kl_div = compute_kl(old_policy, new_policy)
if kl_div > 1.5 * target_kl:
kl_coef *= 2.0 # Policy changing too fast
elif kl_div < target_kl / 1.5:
kl_coef *= 0.5 # Can be more aggressive
# Clip to reasonable bounds
kl_coef = np.clip(kl_coef, 0.0001, 10.0)
```
PPO-Clip (using clipped surrogate objective) has largely replaced adaptive KL in practice due to simpler implementation.