clip model

**CLIP (Contrastive Language-Image Pretraining)** is a **vision-language model trained to align images and text in a shared embedding space** — enabling zero-shot image classification, image search, and serving as the vision backbone of modern generative AI. **How CLIP Works** - **Training Data**: 400M (image, text) pairs scraped from the internet. - **Architecture**: Two encoders — ViT for images, Transformer for text. - **Objective**: Contrastive loss — maximize similarity between correct (image, text) pairs, minimize for incorrect pairs. - **Result**: Images and their descriptions have similar embeddings; unrelated images/texts have dissimilar embeddings. **Zero-Shot Classification** 1. Encode candidate class labels as text: "a photo of a dog", "a photo of a cat". 2. Encode the query image. 3. Find the most similar text embedding → predicted class. 4. No task-specific training required — generalizes to arbitrary categories. **Why CLIP Revolutionized AI** - **Zero-shot transfer**: Competitive with supervised models on 30+ vision benchmarks without task-specific training. - **Universal features**: CLIP embeddings work for retrieval, classification, generation conditioning. - **Stable Diffusion backbone**: CLIP text encoder guides the denoising process in most image generation models. - **Semantic search**: Enables image search by text description (used in Google Photos, Pinterest). **CLIP Variants** - **OpenCLIP**: Open-source CLIP trained on LAION-5B (5 billion pairs). - **SigLIP (Google)**: Sigmoid loss instead of softmax — better performance at smaller batch sizes. - **MetaCLIP**: Meta's CLIP using curated data curation methodology. CLIP is **the foundation of modern vision-language AI** — its shared embedding space enabled the entire ecosystem of multimodal models and controllable image generation.