conservative q-learning

**CQL** (Conservative Q-Learning) is the **leading offline RL algorithm that learns a conservative (lower-bound) Q-function** — penalizing Q-values for out-of-distribution actions to prevent the overestimation that causes standard Q-learning to fail in the offline setting. **CQL Algorithm** - **Conservative Penalty**: Minimize Q-values under the current policy: $min_Q alpha mathbb{E}_{a sim pi}[Q(s,a)]$. - **Data Support**: Maximize Q-values for actions in the dataset: $max_Q mathbb{E}_{a sim D}[Q(s,a)]$. - **Combined**: $L_{CQL} = alpha (mathbb{E}_pi[Q] - mathbb{E}_D[Q]) + L_{TD}$ — pushes down OOD, pushes up in-distribution. - **Lower Bound**: The learned Q-function is a provable lower bound of the true Q-function — conservative but safe. **Why It Matters** - **No Overestimation**: CQL prevents the catastrophic overestimation of out-of-distribution actions that kills standard offline Q-learning. - **Provable**: CQL provides provable lower bounds on the policy's true value — safe deployment guarantees. - **State-of-Art**: CQL is one of the strongest offline RL baselines — works across many domains. **CQL** is **pessimistic Q-learning** — learning a deliberately conservative value function for safe, reliable offline RL.