c51

**C51** (Categorical 51-Atom) is the **first practical distributional RL algorithm** — representing the return distribution as a categorical distribution over 51 equally-spaced atoms, learning the probability of each atom to capture the full distribution of future returns. **C51 Algorithm** - **Atoms**: 51 fixed values $z_i$ equally spaced in $[V_{min}, V_{max}]$ — the support of the distribution. - **Probabilities**: Neural network outputs $p_i(s,a)$ — probability that the return falls in each atom's bin. - **Projection**: After Bellman update, project the shifted distribution back onto the fixed support. - **Loss**: Cross-entropy between the projected target distribution and the predicted distribution. **Why It Matters** - **Breakthrough**: C51 (Bellemare et al., 2017) showed distributional RL works better than expected — not just theoretically interesting. - **Performance**: C51 significantly outperforms standard DQN on Atari — richer gradient signal. - **Foundation**: C51 spawned QR-DQN, IQN, and the distributional RL revolution. **C51** is **the 51-bin histogram of returns** — discretizing the return distribution into 51 atoms for practical distributional reinforcement learning.