Pseudo-Count Methods are exploration techniques that extend count-based exploration to high-dimensional state spaces — using density models to estimate pseudo-counts $hat{N}(s)$ that approximate traditional visit counts, enabling count-based exploration bonuses for complex observations like images.

Pseudo-Count from Density

• Density Model: Train a density model $ ho(s)$ on visited states.
• Pseudo-Count: $hat{N}(s) = frac{ ho(s)(1 - ho'(s))}{ ho'(s) - ho(s)}$ where $ ho'$ is the density after one additional visit.
• Bonus: $r_{bonus} = eta / sqrt{hat{N}(s)}$ — same form as tabular count bonus.
• Models: PixelCNN, context tree switching, or other generative models for density estimation.

Why It Matters

• High-Dimensional: Extends count-based exploration to pixel observations — where tabular counts are infeasible.
• Theory Meets Practice: Bridges the theoretical elegance of count-based exploration with practical deep RL.
• Montezuma: Pseudo-counts enabled early progress on hard-exploration Atari games.

Pseudo-Count is counting in pixel space — using density models to approximate visit counts for scalable count-based exploration.