LLaVA (Large Language-and-Vision Assistant) is the pioneering open-source vision-language model that introduced visual instruction tuning — connecting a CLIP vision encoder to a LLaMA/Vicuna language model and training on GPT-4-generated visual conversation data to create a multimodal assistant that can describe images, answer visual questions, reason about visual content, and follow complex instructions involving both text and images.

What Is LLaVA?

• Definition: A multimodal model (from University of Wisconsin-Madison and Microsoft Research, 2023) that combines a pretrained CLIP ViT-L/14 vision encoder with a pretrained LLaMA/Vicuna language model through a trainable projection layer — fine-tuned on 158K visual instruction-following examples generated by GPT-4.
• Visual Instruction Tuning: The key innovation — using GPT-4 (text-only) to generate high-quality conversation, detailed description, and complex reasoning data about images (using image captions and bounding boxes as input to GPT-4), then training the multimodal model on this synthetic data.
• Architecture: CLIP ViT-L/14 encodes the image into patch embeddings → a linear projection (LLaVA 1.0) or MLP projection (LLaVA 1.5) maps visual tokens to the LLM's embedding space → visual tokens are concatenated with text tokens → the LLM generates the response.
• LLaVA 1.5: The improved version that replaced the linear projection with a 2-layer MLP, used higher resolution (336×336), and trained on 665K visual instruction examples — achieving state-of-the-art results on 11 benchmarks with a simple, reproducible architecture.

LLaVA Model Versions

Why LLaVA Matters

• Simplicity: LLaVA's architecture is remarkably simple — a vision encoder, a projection layer, and an LLM. No complex cross-attention modules, no additional encoders. This simplicity made it reproducible and extensible.
• Data-Centric Innovation: The breakthrough was the training data, not the architecture — using GPT-4 to generate visual instruction data showed that synthetic data quality matters more than architectural complexity.
• Open-Source Standard: LLaVA became the reference architecture for open-source VLMs — most subsequent models (InternVL, Cambrian, LLaVA-NeXT) follow the same encoder-projector-LLM pattern.
• Community Impact: Fully open-source (code, data, weights) — spawned hundreds of derivative models, fine-tunes, and research papers building on the LLaVA architecture.

LLaVA is the open-source vision-language model that established visual instruction tuning as the standard approach for building multimodal AI assistants — demonstrating that connecting a CLIP vision encoder to an LLM through a simple projection layer, trained on GPT-4-generated visual conversation data, produces powerful multimodal capabilities that rival proprietary systems.