bit diffusion

**Bit Diffusion** is a **diffusion model variant that represents discrete data as binary (bit) vectors and applies continuous diffusion in the binary representation space** — encoding each discrete token as a set of bits, then treating each bit as a continuous variable for standard Gaussian diffusion. **Bit Diffusion Approach** - **Binary Encoding**: Convert discrete tokens to binary vectors — e.g., token ID 42 → [1,0,1,0,1,0,...]. - **Analog Bits**: Treat binary values as continuous — relax {0,1} to continuous values in [0,1] or ℝ. - **Gaussian Diffusion**: Apply standard continuous diffusion to the analog bit vectors — add and remove Gaussian noise. - **Rounding**: At generation time, round continuous values back to binary — decode to discrete tokens. **Why It Matters** - **Best of Both**: Combines the simplicity of continuous Gaussian diffusion with discrete output generation. - **Image Generation**: Originally proposed for discrete image generation — pixel values as bit sequences. - **Scalability**: Leverages the well-developed toolkit of continuous diffusion models for discrete problems. **Bit Diffusion** is **treating bits as continuous signals** — encoding discrete data in binary and applying standard Gaussian diffusion for generation.