interleaved image-text generation

**Interleaved Image-Text Generation** is the **process of generating coherent sequences containing both text and images** — enabling models to write illustrated articles, create instructional manuals with diagrams, or tell visual stories that flow naturally between modalities. **What Is Interleaved Generation?** - **Definition**: Output stream contains sequence of $[T_1, T_2, I_1, T_3, I_2, ...]$. - **Contrast**: Most models are "Text-to-Image" (generating one image) or "Image-to-Text" (captioning). Interleaved models do both continuously. - **Models**: CM3, MM-Interleaved, GPT-4V (in principle), Gemini. **Why It Matters** - **Rich Communication**: Humans naturally mix speech, gesture, and showing objects; AI should too. - **Storytelling**: Can generate a children's book with consistent characters and plot. - **Documentation**: Automatically generating "How-To" guides with screenshots inserted at the right steps. **Technical Challenges** - **Modality Gap**: Aligning the vector space of text tokens and image pixels/tokens. - **Coherence**: Ensuring the image $I_2$ is consistent with the text $T_1$ and previous image $I_1$. - **Tokenization**: Requires efficient visual tokenizers (like VQ-VAE) to treat images as "words" in the vocabulary. **Interleaved Image-Text Generation** is **the future of automated content creation** — moving beyond static media to dynamic, multi-modal narratives.