CLIP (Contrastive Language-Image Pre-training) aligns text and image embeddings for zero-shot visual understanding. Approach: Train image encoder and text encoder jointly such that matching image-text pairs have similar embeddings, non-matching pairs have different embeddings. Contrastive learning across modalities. Training data: 400M image-text pairs from internet (WebImageText dataset). Scale is key. Architecture: Image encoder (ViT or ResNet), text encoder (Transformer), learned projection to shared embedding space, contrastive loss over batch. Zero-shot inference: Encode class names as text ("a photo of a {class}"), encode image, classify by highest similarity to text embeddings. Prompt engineering: "A photo of a {class}" works better than just class name. Prompt ensembling improves results. Capabilities: Zero-shot classification, image-text retrieval, supports many visual tasks without task-specific training. Limitations: Struggles with fine-grained categories, counting, spatial relationships. Impact: Foundation for many multimodal models, text-conditional image generation (DALL-E, Stable Diffusion use CLIP), revolutionized zero-shot visual recognition.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.