<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Multimodal AI — Vision + Language + Audio</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">encode each modality into a shared embedding space, then let the LLM reason over all of them together</text>

  <!-- === TOP: Architecture === -->
  <rect x="25" y="62" width="710" height="148" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="82" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Multimodal Model Architecture</text>

  <!-- Vision encoder -->
  <rect x="40" y="95" width="100" height="55" rx="5" fill="#0f1a2a" stroke="#60a5fa" stroke-width="1"/>
  <text x="90" y="115" fill="#93c5fd" font-size="9" text-anchor="middle" font-weight="600">Vision Encoder</text>
  <text x="90" y="129" fill="#8b98a5" font-size="8" text-anchor="middle">ViT / SigLIP</text>
  <text x="90" y="141" fill="#6b7684" font-size="7.5" text-anchor="middle">patch → embeddings</text>

  <!-- Audio encoder -->
  <rect x="40" y="158" width="100" height="42" rx="5" fill="#14261f" stroke="#34d399" stroke-width="1"/>
  <text x="90" y="176" fill="#6ee7b7" font-size="9" text-anchor="middle" font-weight="600">Audio Encoder</text>
  <text x="90" y="190" fill="#8b98a5" font-size="8" text-anchor="middle">Whisper / mel spec</text>

  <!-- Projection / adapter -->
  <path d="M143,122 L185,140" fill="none" stroke="#60a5fa" stroke-width="0.8"/>
  <path d="M143,178 L185,155" fill="none" stroke="#34d399" stroke-width="0.8"/>
  <rect x="188" y="128" width="80" height="40" rx="4" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="228" y="147" fill="#fbbf24" font-size="8.5" text-anchor="middle" font-weight="600">Projection</text>
  <text x="228" y="161" fill="#a1701a" font-size="7.5" text-anchor="middle">align to LLM dim</text>

  <!-- Text tokenizer -->
  <rect x="188" y="95" width="80" height="28" rx="3" fill="#1a1520" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="228" y="113" fill="#c4b5fd" font-size="8.5" text-anchor="middle">Text tokens</text>

  <!-- Arrows into LLM -->
  <path d="M270,108 L310,130" fill="none" stroke="#a78bfa" stroke-width="0.7"/>
  <path d="M270,148 L310,148" fill="none" stroke="#f59e0b" stroke-width="0.7"/>

  <!-- LLM backbone -->
  <rect x="315" y="100" width="150" height="85" rx="8" fill="#1a1520" stroke="#a78bfa" stroke-width="1.3"/>
  <text x="390" y="125" fill="#c4b5fd" font-size="11" text-anchor="middle" font-weight="600">LLM Backbone</text>
  <text x="390" y="142" fill="#8b98a5" font-size="8.5" text-anchor="middle">Llama / Gemma / Qwen</text>
  <text x="390" y="157" fill="#8b98a5" font-size="8" text-anchor="middle">interleaved modality tokens</text>
  <text x="390" y="172" fill="#6b7684" font-size="7.5" text-anchor="middle">autoregressive generation</text>

  <!-- Output -->
  <path d="M468,142 L510,142" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="508,139 514,142 508,145" fill="#8b98a5"/>

  <rect x="518" y="100" width="105" height="35" rx="4" fill="#14261f" stroke="#34d399" stroke-width="0.9"/>
  <text x="570" y="120" fill="#6ee7b7" font-size="8.5" text-anchor="middle">Text output</text>

  <rect x="518" y="142" width="105" height="35" rx="4" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.9"/>
  <text x="570" y="162" fill="#93c5fd" font-size="8.5" text-anchor="middle">Image gen (optional)</text>

  <!-- Annotation -->
  <text x="660" y="115" fill="#6b7684" font-size="8">can understand</text>
  <text x="660" y="128" fill="#6b7684" font-size="8">and generate</text>
  <text x="660" y="141" fill="#6b7684" font-size="8">across modalities</text>

  <!-- === MIDDLE LEFT: Training approaches === -->
  <rect x="25" y="218" width="350" height="110" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="236" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Training Approaches</text>

  <text x="45" y="256" fill="#60a5fa" font-size="8.5" font-weight="600">Contrastive (CLIP)</text>
  <text x="175" y="256" fill="#8b98a5" font-size="8.5">align image↔text embeddings</text>
  <text x="45" y="272" fill="#34d399" font-size="8.5" font-weight="600">Generative (LLaVA)</text>
  <text x="175" y="272" fill="#8b98a5" font-size="8.5">vision tokens → LLM next-token loss</text>
  <text x="45" y="288" fill="#fbbf24" font-size="8.5" font-weight="600">Interleaved (Gemini)</text>
  <text x="180" y="288" fill="#8b98a5" font-size="8.5">native multi-modal pre-training</text>
  <text x="45" y="304" fill="#c4b5fd" font-size="8.5" font-weight="600">Diffusion (DALL-E)</text>
  <text x="170" y="304" fill="#8b98a5" font-size="8.5">text-conditioned image generation</text>
  <text x="45" y="320" fill="#f87171" font-size="8.5" font-weight="600">Any-to-Any (GPT-4o)</text>
  <text x="185" y="320" fill="#8b98a5" font-size="8.5">input/output all modalities natively</text>

  <!-- === MIDDLE RIGHT: Modality details === -->
  <rect x="390" y="218" width="345" height="110" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="562" y="236" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Modality Encoding</text>

  <text x="410" y="256" fill="#60a5fa" font-size="8.5" font-weight="600">Image:</text>
  <text x="455" y="256" fill="#8b98a5" font-size="8.5">224-1024px → 16×16 patches → ViT</text>
  <text x="410" y="272" fill="#34d399" font-size="8.5" font-weight="600">Video:</text>
  <text x="455" y="272" fill="#8b98a5" font-size="8.5">sample frames → per-frame ViT + temporal</text>
  <text x="410" y="288" fill="#fbbf24" font-size="8.5" font-weight="600">Audio:</text>
  <text x="455" y="288" fill="#8b98a5" font-size="8.5">mel spectrogram → conv encoder (Whisper)</text>
  <text x="410" y="304" fill="#c4b5fd" font-size="8.5" font-weight="600">3D/Point:</text>
  <text x="465" y="304" fill="#8b98a5" font-size="8.5">point cloud → transformer encoder</text>
  <text x="410" y="320" fill="#8b98a5" font-size="8.5" font-weight="600">Document:</text>
  <text x="475" y="320" fill="#8b98a5" font-size="8.5">OCR-free: image patches directly</text>

  <!-- === BOTTOM: Models === -->
  <rect x="25" y="338" width="710" height="68" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="356" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Frontier Multimodal Models</text>

  <text x="85" y="378" fill="#c4b5fd" font-size="9.5" text-anchor="middle" font-weight="600">GPT-4o</text>
  <text x="85" y="392" fill="#8b98a5" font-size="8" text-anchor="middle">any-to-any native</text>

  <text x="210" y="378" fill="#fbbf24" font-size="9.5" text-anchor="middle" font-weight="600">Gemini 2.0</text>
  <text x="210" y="392" fill="#8b98a5" font-size="8" text-anchor="middle">natively multimodal</text>

  <text x="335" y="378" fill="#34d399" font-size="9.5" text-anchor="middle" font-weight="600">Claude 3.5</text>
  <text x="335" y="392" fill="#8b98a5" font-size="8" text-anchor="middle">vision + documents</text>

  <text x="460" y="378" fill="#60a5fa" font-size="9.5" text-anchor="middle" font-weight="600">LLaVA-NeXT</text>
  <text x="460" y="392" fill="#8b98a5" font-size="8" text-anchor="middle">open-source VLM</text>

  <text x="585" y="378" fill="#f87171" font-size="9.5" text-anchor="middle" font-weight="600">Qwen2-VL</text>
  <text x="585" y="392" fill="#8b98a5" font-size="8" text-anchor="middle">dynamic resolution</text>

  <text x="695" y="378" fill="#8b98a5" font-size="9.5" text-anchor="middle" font-weight="600">Pixtral</text>
  <text x="695" y="392" fill="#8b98a5" font-size="8" text-anchor="middle">Mistral vision</text>

  <!-- Key insight -->
  <rect x="25" y="414" width="710" height="22" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="429" fill="#fbbf24" font-size="9" text-anchor="middle">The key insight: once encoded into the same vector space, the LLM treats vision/audio tokens the same as text tokens.</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Multimodal models unify perception — one model sees, hears, reads, and reasons across all inputs simultaneously.</text>
</svg>
```e Vision Transformer (ViT)** showed that the Transformer architecture built for language works just as well on images, and that insight is the bridge to today's multimodal models. Instead of processing pixels with convolutions, a ViT cuts an image into a grid of small patches, treats each patch as a token, and feeds the sequence into a standard Transformer encoder. Once an image is "just a sequence of tokens," it can share an architecture — and eventually a single model — with text, which is exactly what vision-language and multimodal systems exploit.\n\n```svg\n<svg viewBox="0 0 820 486" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">\n  <rect x="0" y="0" width="820" height="486" fill="#0d1117"/>\n  <text x="410" y="34" fill="#e6edf3" font-size="22" font-weight="700" text-anchor="middle">Vision Transformers &amp; Multimodal Models — Seeing with a Transformer</text>\n  <text x="410" y="58" fill="#8b98a5" font-size="13" text-anchor="middle">cut an image into patches, treat them as tokens — the same trick that lets one model jointly understand images and text</text>\n  <rect x="40" y="88" width="372" height="290" rx="8" fill="#0b1220" stroke="#233043" stroke-width="1.2"/>\n  <text x="58" y="114" fill="#cbd5e1" font-size="13" font-weight="700">ViT: an image becomes a sequence of tokens</text>\n  <rect x="64" y="132" width="20" height="20" fill="#1e3a5f" stroke="#38bdf8" stroke-width="0.8"/>\n  <rect x="86" y="132" width="20" height="20" fill="#2a4a6f" stroke="#38bdf8" stroke-width="0.8"/>\n  <rect x="108" y="132" width="20" height="20" fill="#1e3a5f" stroke="#38bdf8" stroke-width="0.8"/>\n  <rect x="64" y="154" width="20" height="20" fill="#2a4a6f" stroke="#38bdf8" stroke-width="0.8"/>\n  <rect x="86" y="154" width="20" height="20" fill="#38607f" stroke="#38bdf8" stroke-width="0.8"/>\n  <rect x="108" y="154" width="20" height="20" fill="#2a4a6f" stroke="#38bdf8" stroke-width="0.8"/>\n  <rect x="64" y="176" width="20" height="20" fill="#1e3a5f" stroke="#38bdf8" stroke-width="0.8"/>\n  <rect x="86" y="176" width="20" height="20" fill="#2a4a6f" stroke="#38bdf8" stroke-width="0.8"/>\n  <rect x="108" y="176" width="20" height="20" fill="#1e3a5f" stroke="#38bdf8" stroke-width="0.8"/>\n  <text x="96.0" y="214" fill="#7dd3fc" font-size="12" text-anchor="middle">split into patches</text>\n  <rect x="158" y="144" width="42" height="30" rx="5" fill="#0f1c17" stroke="#34d399" stroke-width="1.2"/>\n  <text x="179" y="164" fill="#6ee7b7" font-size="13" text-anchor="middle">CLS</text>\n  <rect x="205" y="144" width="42" height="30" rx="5" fill="#101826" stroke="#38bdf8" stroke-width="1.2"/>\n  <text x="226" y="164" fill="#7dd3fc" font-size="13" text-anchor="middle">1</text>\n  <rect x="252" y="144" width="42" height="30" rx="5" fill="#101826" stroke="#38bdf8" stroke-width="1.2"/>\n  <text x="273" y="164" fill="#7dd3fc" font-size="13" text-anchor="middle">2</text>\n  <rect x="299" y="144" width="42" height="30" rx="5" fill="#101826" stroke="#38bdf8" stroke-width="1.2"/>\n  <text x="320" y="164" fill="#7dd3fc" font-size="13" text-anchor="middle">3</text>\n  <rect x="346" y="144" width="42" height="30" rx="5" fill="#101826" stroke="#38bdf8" stroke-width="1.2"/>\n  <text x="367" y="164" fill="#7dd3fc" font-size="13" text-anchor="middle">4</text>\n  <text x="278" y="136" fill="#8b98a5" font-size="12" text-anchor="middle">patch tokens + a [CLS] token</text>\n  <line x1="132" y1="165.0" x2="152" y2="159" stroke="#38bdf8" stroke-width="1.6"/>\n  <path d="M145 165 L152 159 L143 158" stroke="#38bdf8" stroke-width="1.6" fill="none"/>\n  <rect x="64" y="198" width="324" height="28" rx="6" fill="#151b23" stroke="#6b7684" stroke-width="1.1"/>\n  <text x="226.0" y="216" fill="#c9d4e0" font-size="12" text-anchor="middle">+ linear embedding &#38; positional encoding</text>\n  <line x1="226.0" y1="174" x2="226.0" y2="196" stroke="#3f5169" stroke-width="1.6"/>\n  <path d="M222 188 L226 196 L230 188" stroke="#3f5169" stroke-width="1.6" fill="none"/>\n  <rect x="64" y="244" width="324" height="46" rx="8" fill="#2a1420" stroke="#fbbf24" stroke-width="1.6"/>\n  <text x="226.0" y="265" fill="#fcd34d" font-size="12.5" font-weight="700" text-anchor="middle">Transformer Encoder</text>\n  <text x="226.0" y="282" fill="#8b98a5" font-size="12" text-anchor="middle">self-attention lets every patch see every other patch</text>\n  <line x1="226.0" y1="226" x2="226.0" y2="242" stroke="#3f5169" stroke-width="1.6"/>\n  <path d="M222 234 L226 242 L230 234" stroke="#3f5169" stroke-width="1.6" fill="none"/>\n  <rect x="130" y="314" width="192" height="40" rx="8" fill="#0f1c17" stroke="#34d399" stroke-width="1.6"/>\n  <text x="226.0" y="332" fill="#6ee7b7" font-size="13" font-weight="700" text-anchor="middle">class label / image features</text>\n  <text x="226.0" y="347" fill="#8b98a5" font-size="13" text-anchor="middle">no convolutions — but needs large-scale pre-training</text>\n  <line x1="226.0" y1="290" x2="226.0" y2="312" stroke="#34d399" stroke-width="1.6"/>\n  <path d="M222 304 L226 312 L230 304" stroke="#34d399" stroke-width="1.6" fill="none"/>\n  <rect x="432" y="88" width="348" height="290" rx="8" fill="#0b1220" stroke="#233043" stroke-width="1.2"/>\n  <text x="450" y="114" fill="#cbd5e1" font-size="13" font-weight="700">From image tokens to multimodal</text>\n  <text x="452" y="136" fill="#93c5fd" font-size="12" font-weight="700" transform="rotate(-90 462 172)">image emb</text>\n  <rect x="502" y="132" width="18" height="18" fill="#0f2a1e" stroke="#34d399" stroke-width="1"/>\n  <text x="511.0" y="145" fill="#6ee7b7" font-size="13" text-anchor="middle">&#10003;</text>\n  <rect x="522" y="132" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="542" y="132" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="562" y="132" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="502" y="152" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="522" y="152" width="18" height="18" fill="#0f2a1e" stroke="#34d399" stroke-width="1"/>\n  <text x="531.0" y="165" fill="#6ee7b7" font-size="13" text-anchor="middle">&#10003;</text>\n  <rect x="542" y="152" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="562" y="152" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="502" y="172" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="522" y="172" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="542" y="172" width="18" height="18" fill="#0f2a1e" stroke="#34d399" stroke-width="1"/>\n  <text x="551.0" y="185" fill="#6ee7b7" font-size="13" text-anchor="middle">&#10003;</text>\n  <rect x="562" y="172" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="502" y="192" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="522" y="192" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="542" y="192" width="18" height="18" fill="#141b23" stroke="#2b3a4d" stroke-width="1"/>\n  <rect x="562" y="192" width="18" height="18" fill="#0f2a1e" stroke="#34d399" stroke-width="1"/>\n  <text x="571.0" y="205" fill="#6ee7b7" font-size="13" text-anchor="middle">&#10003;</text>\n  <text x="542" y="227" fill="#93c5fd" font-size="12" text-anchor="middle">text emb</text>\n  <text x="618" y="146" fill="#cbd5e1" font-size="13" font-weight="700">CLIP</text>\n  <text x="618" y="164" fill="#8b98a5" font-size="12">contrastive training aligns</text>\n  <text x="618" y="178" fill="#8b98a5" font-size="12">matching image–text pairs</text>\n  <text x="618" y="192" fill="#8b98a5" font-size="12">on the diagonal → zero-shot</text>\n  <line x1="450" y1="238" x2="762" y2="238" stroke="#233043" stroke-width="1"/>\n  <text x="450" y="260" fill="#a78bfa" font-size="12" font-weight="700">VLM: give an LLM eyes</text>\n  <rect x="450" y="276" width="86" height="42" rx="8" fill="#101826" stroke="#38bdf8" stroke-width="1.6"/>\n  <text x="493" y="296" fill="#7dd3fc" font-size="12" text-anchor="middle">vision</text>\n  <text x="493" y="310" fill="#7dd3fc" font-size="12" text-anchor="middle">encoder (ViT)</text>\n  <rect x="552" y="276" width="60" height="42" rx="8" fill="#1a1226" stroke="#c084fc" stroke-width="1.6"/>\n  <text x="582" y="301" fill="#d8b4fe" font-size="12" text-anchor="middle">projector</text>\n  <rect x="628" y="276" width="60" height="42" rx="8" fill="#2a1420" stroke="#fbbf24" stroke-width="1.6"/>\n  <text x="658" y="301" fill="#fcd34d" font-size="13" font-weight="700" text-anchor="middle">LLM</text>\n  <rect x="704" y="276" width="58" height="42" rx="8" fill="#0f1c17" stroke="#34d399" stroke-width="1.6"/>\n  <text x="733" y="301" fill="#6ee7b7" font-size="12" text-anchor="middle">answer</text>\n  <line x1="536" y1="297" x2="552" y2="297" stroke="#3f5169" stroke-width="1.5"/>\n  <path d="M544 301 L552 297 L544 293" stroke="#3f5169" stroke-width="1.5" fill="none"/>\n  <line x1="612" y1="297" x2="628" y2="297" stroke="#3f5169" stroke-width="1.5"/>\n  <path d="M620 301 L628 297 L620 293" stroke="#3f5169" stroke-width="1.5" fill="none"/>\n  <line x1="688" y1="297" x2="704" y2="297" stroke="#3f5169" stroke-width="1.5"/>\n  <path d="M696 301 L704 297 L696 293" stroke="#3f5169" stroke-width="1.5" fill="none"/>\n  <rect x="612" y="332" width="92" height="24" rx="5" fill="#151b23" stroke="#6b7684" stroke-width="1"/>\n  <text x="658" y="348" fill="#c9d4e0" font-size="12" text-anchor="middle">text prompt</text>\n  <line x1="658" y1="332" x2="658" y2="320" stroke="#3f5169" stroke-width="1.4"/>\n  <path d="M662 328 L658 320 L654 328" stroke="#3f5169" stroke-width="1.4" fill="none"/>\n  <rect x="40" y="392" width="740" height="70" rx="8" fill="#0d1117" stroke="#38bdf8" stroke-width="1.3"/>\n  <text x="58" y="420" fill="#7dd3fc" font-size="12.5" font-weight="700">One unifying idea</text>\n  <text x="200" y="420" fill="#e6edf3" font-size="12.5">image patches, words — even audio frames — all become tokens in one Transformer.</text>\n  <text x="58" y="444" fill="#8b98a5" font-size="12">That shared token space is why a single architecture can see, read, and — in “omni” models — map any modality to any other.</text>\n</svg>\n```\n\n**A ViT turns an image into patch tokens.** The image is split into fixed-size patches (often 16×16 pixels), each patch is flattened and linearly projected into an embedding, and learned positional encodings are added so the model knows where each patch sat. A special classification token is prepended, the whole sequence runs through Transformer encoder layers where self-attention lets every patch attend to every other, and the output at the classification token is used to predict the label. There are no convolutions anywhere in the core model.\n\n**ViT trades inductive bias for scale.** Convolutional networks bake in helpful assumptions — locality and translation equivariance — that ViTs lack, so on small datasets a ViT actually underperforms a comparable CNN. Its advantage appears with scale: pre-trained on very large image collections, a ViT matches or beats the best CNNs, because attention can learn flexible, long-range relationships that convolutions cannot. Data-efficient training recipes and distillation later narrowed the data requirement.\n\n**CLIP aligns vision and language in a shared space.** Trained contrastively on hundreds of millions of image–caption pairs, CLIP pairs an image encoder (usually a ViT) with a text encoder and pushes matching image–text embeddings together while pushing mismatched ones apart. The result is a joint embedding space where an image and its description land near each other, enabling zero-shot classification and image–text retrieval without task-specific training. CLIP's image encoder became the visual front-end for much of what followed.\n\n**Vision-language models give a language model eyes.** Systems such as LLaVA, Flamingo, and GPT-4V connect a pretrained vision encoder to a large language model through a small projection or adapter, so image-derived tokens enter the LLM's context alongside the text prompt. The LLM can then answer questions about a picture, read documents, or describe scenes. "Omni" or any-to-any models push this further, mapping among text, images, audio, and video within one model, so a single system can both perceive and generate across modalities.\n\n**The payoff and the open problems.** Tokenizing every modality unifies perception and language under one Transformer, which is why progress in one area now lifts the others, and why frontier assistants are natively multimodal. The hard parts are the cost of high-resolution and video inputs, hallucination on fine visual detail, and the resolution-versus-token-count trade-off — more patches mean sharper vision but a longer, more expensive sequence. Better visual tokenization and grounding are where much of the current research sits.\n\n| Stage | What it does | Key idea |\n|---|---|---|\n| Vision Transformer | image → patch tokens → encoder | patches are tokens |\n| CLIP | align image and text embeddings | one contrastive shared space |\n| Vision-language model | vision encoder feeds an LLM | image tokens in the LLM's context |\n| Omni / any-to-any | map among many modalities | one model perceives and generates |\n\nRead vision transformers and multimodal models through a *tokenize-everything* lens rather than a *new-vision-network* lens: the breakthrough is not a better image classifier but the realization that once patches, words, and audio frames are all tokens, one Transformer can attend across them — turning separate vision and language systems into a single model that sees and reads at once.\n