pseudo-count methods
**Pseudo-Count Methods** are **exploration techniques that extend count-based exploration to high-dimensional state spaces** — using density models to estimate pseudo-counts $hat{N}(s)$ that approximate traditional visit counts, enabling count-based exploration bonuses for complex observations like images.
**Pseudo-Count from Density**
- **Density Model**: Train a density model $
ho(s)$ on visited states.
- **Pseudo-Count**: $hat{N}(s) = frac{
ho(s)(1 -
ho'(s))}{
ho'(s) -
ho(s)}$ where $
ho'$ is the density after one additional visit.
- **Bonus**: $r_{bonus} = eta / sqrt{hat{N}(s)}$ — same form as tabular count bonus.
- **Models**: PixelCNN, context tree switching, or other generative models for density estimation.
**Why It Matters**
- **High-Dimensional**: Extends count-based exploration to pixel observations — where tabular counts are infeasible.
- **Theory Meets Practice**: Bridges the theoretical elegance of count-based exploration with practical deep RL.
- **Montezuma**: Pseudo-counts enabled early progress on hard-exploration Atari games.
**Pseudo-Count** is **counting in pixel space** — using density models to approximate visit counts for scalable count-based exploration.