clip (contrastive language-image pre-training)

CLIP (Contrastive Language-Image Pre-training) aligns text and image embeddings for zero-shot visual understanding. **Approach**: Train image encoder and text encoder jointly such that matching image-text pairs have similar embeddings, non-matching pairs have different embeddings. Contrastive learning across modalities. **Training data**: 400M image-text pairs from internet (WebImageText dataset). Scale is key. **Architecture**: Image encoder (ViT or ResNet), text encoder (Transformer), learned projection to shared embedding space, contrastive loss over batch. **Zero-shot inference**: Encode class names as text ("a photo of a {class}"), encode image, classify by highest similarity to text embeddings. **Prompt engineering**: "A photo of a {class}" works better than just class name. Prompt ensembling improves results. **Capabilities**: Zero-shot classification, image-text retrieval, supports many visual tasks without task-specific training. **Limitations**: Struggles with fine-grained categories, counting, spatial relationships. **Impact**: Foundation for many multimodal models, text-conditional image generation (DALL-E, Stable Diffusion use CLIP), revolutionized zero-shot visual recognition.