quantile regression dqn

**QR-DQN** (Quantile Regression DQN) is a **distributional RL algorithm that learns quantiles of the return distribution** — instead of fixed atoms (like C51), QR-DQN directly learns the values at fixed quantile levels using quantile regression, providing a flexible, non-parametric representation. **QR-DQN Algorithm** - **Quantiles**: Learn $N$ quantile values $ heta_i(s,a)$ at fixed quantile levels $ au_i = (2i-1)/(2N)$ for $i = 1,...,N$. - **Loss**: Quantile Huber loss — asymmetric loss that penalizes over/under-estimation differently for each quantile. - **No Projection**: Unlike C51, no need to project distributions onto a fixed support — quantiles are free-form. - **Q-Value**: $Q(s,a) = frac{1}{N}sum_i heta_i(s,a)$ — the mean of the quantile values. **Why It Matters** - **Flexible**: Quantiles can represent any distribution shape — not limited to a fixed support like C51. - **Simpler**: No distribution projection needed — cleaner algorithm than C51. - **Risk**: Different quantiles enable risk-sensitive policies — optimize for extreme quantiles (CVaR). **QR-DQN** is **learning the quantiles of returns** — a flexible, projection-free distributional RL method using quantile regression.