clip

**CLIP (Contrastive Language-Image Pretraining)** is a **multimodal AI model that learns to align images and text in a shared embedding space through contrastive learning** — training dual encoders (a Vision Transformer for images and a text Transformer for captions) on 400 million image-text pairs from the web to learn visual concepts from natural language supervision, enabling zero-shot image classification, text-to-image search, and cross-modal retrieval without task-specific training data. **What Is CLIP?** - **Definition**: A vision-language model developed by OpenAI that jointly trains an image encoder and a text encoder to produce embeddings in a shared 512-dimensional vector space — images and their matching text descriptions are mapped to nearby points, while non-matching pairs are pushed apart using contrastive loss (InfoNCE). - **Dual Encoder Architecture**: The image encoder (Vision Transformer ViT-B/32, ViT-L/14, or ResNet variants) processes images into embedding vectors — the text encoder (12-layer Transformer) processes text into embedding vectors in the same space. Similarity is computed as cosine distance between embeddings. - **Contrastive Training**: Given a batch of N image-text pairs, CLIP maximizes the cosine similarity of the N correct pairs while minimizing similarity of the N²-N incorrect pairs — learning to match images with their descriptions rather than predicting fixed class labels. - **Web-Scale Data**: Trained on 400M image-text pairs collected from the internet (WebImageText dataset) — the scale and diversity of web data enables CLIP to learn robust visual concepts that transfer across domains without fine-tuning. **Why CLIP Matters** - **Zero-Shot Classification**: CLIP classifies images into arbitrary categories without training examples — encode class names as text prompts ("a photo of a dog"), compute similarity with the image embedding, and select the highest-scoring class. Achieves competitive accuracy with supervised models on many benchmarks. - **Foundation for Generative AI**: CLIP embeddings guide text-to-image generation in Stable Diffusion, DALL-E, and other diffusion models — the text encoder provides the conditioning signal that steers image generation toward the text prompt. - **Universal Image Search**: CLIP enables searching image databases using natural language queries — encode the query as text, find images with the most similar embeddings, enabling semantic search that understands concepts rather than just matching keywords. - **Prompt Engineering**: Classification accuracy depends on prompt format — "a photo of a {class}" works better than just "{class}" because it matches the distribution of web captions CLIP was trained on. **CLIP Applications** - **Image Classification**: Zero-shot classification on ImageNet, CIFAR, and domain-specific datasets without fine-tuning. - **Image Search**: Natural language search over image databases — "sunset over mountains" finds relevant images by embedding similarity. - **Content Moderation**: Detect inappropriate content by computing similarity with text descriptions of policy violations. - **Product Matching**: Match product photos to catalog descriptions in e-commerce applications. - **Accessibility**: Generate image descriptions for visually impaired users by finding the most similar text descriptions. | CLIP Variant | Image Encoder | Embedding Dim | ImageNet Zero-Shot | Parameters | |-------------|--------------|--------------|-------------------|-----------| | CLIP ViT-B/32 | ViT-Base, patch 32 | 512 | 63.2% | 151M | | CLIP ViT-B/16 | ViT-Base, patch 16 | 512 | 68.3% | 150M | | CLIP ViT-L/14 | ViT-Large, patch 14 | 768 | 75.3% | 428M | | CLIP RN50 | ResNet-50 | 1024 | 58.2% | 102M | | OpenCLIP ViT-G/14 | ViT-Giant | 1024 | 80.1% | 1.8B | **CLIP is the foundational vision-language model that revolutionized multimodal AI** — demonstrating that contrastive learning on web-scale image-text data enables robust zero-shot visual understanding, powering image search, content moderation, and serving as the text encoder backbone for modern text-to-image generation systems.