distributional rl

**Distributional RL** is a **reinforcement learning framework that learns the full distribution of returns (not just the expected value)** — instead of estimating $Q(s,a) = mathbb{E}[R]$, distributional RL estimates the random variable $Z(s,a)$ such that $Q(s,a) = mathbb{E}[Z(s,a)]$. **Distributional RL Methods** - **C51**: Represent the distribution as a categorical distribution over 51 atoms (fixed support). - **QR-DQN**: Learn quantiles of the return distribution — flexible, non-parametric. - **IQN**: Implicit Quantile Network — sample any quantile at inference time. - **Bellman Update**: $Z(s,a) overset{D}{=} R + gamma Z(s', a')$ — distribute the Bellman equation over the full distribution. **Why It Matters** - **Richer Information**: The full distribution captures risk, multimodality, and uncertainty — not just the mean. - **Better Optimization**: Learning distributions provides richer gradient signals — improves optimization. - **Risk-Sensitive**: Enables risk-sensitive policies — optimize for different quantiles (worst-case, median, etc.). **Distributional RL** is **learning the entire story of returns** — capturing the full distribution of outcomes for richer, risk-aware reinforcement learning.