llava (large language and vision assistant)

**LLaVA** (Large Language and Vision Assistant) is an **open-source multimodal model** — that combines a vision encoder (CLIP ViT-L) with an LLM (Vicuna/LLaMA) to creating a "visual chatbot" with capabilities similar to GPT-4 Vision. **What Is LLaVA?** - **Definition**: End-to-end trained large multimodal model. - **Architecture**: Simple projection layer connects CLIP (frozen) to LLaMA (fine-tuned). - **Data Innovation**: Used GPT-4 (text-only) to generate multimodal instruction-following data from image captions and bounding boxes. - **Philosophy**: Simple architecture + High-quality instruction data = SOTA performance. **Why LLaVA Matters** - **Simplicity**: Unlike the complex Q-Former of BLIP-2, LLaVA just uses a linear projection (MLP). - **Open Source**: The code, data, and weights are fully open, driving the open VLM community. - **Science QA**: Achieved state-of-the-art on reasoning benchmarks. **Training Stages** 1. **Feature Alignment**: Pre-training to align image features to word embeddings. 2. **Visual Instruction Tuning**: Fine-tuning on the GPT-4 generated instruction data (conversations, reasoning). **LLaVA** is **the "Hello World" of modern VLMs** — its simple, effective recipe became the standard basline for nearly all subsequent open-source multimodal research.