CLIP (Contrastive Language-Image Pretraining) is a multimodal AI model that learns to align images and text in a shared embedding space through contrastive learning — training dual encoders (a Vision Transformer for images and a text Transformer for captions) on 400 million image-text pairs from the web to learn visual concepts from natural language supervision, enabling zero-shot image classification, text-to-image search, and cross-modal retrieval without task-specific training data.

What Is CLIP?

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">CLIP — Contrastive Language-Image Pretraining</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">dual encoder learns shared embedding space for images and text via contrastive loss</text>

  <!-- Main architecture -->
  <rect x="30" y="65" width="700" height="230" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>

  <!-- Image encoder (left) -->
  <rect x="55" y="85" width="160" height="190" rx="6" fill="#0b1220" stroke="#60a5fa" stroke-width="1.2"/>
  <text x="135" y="105" fill="#93c5fd" font-size="10" font-weight="600" text-anchor="middle">Image Encoder</text>
  <text x="135" y="120" fill="#6b7684" font-size="8" text-anchor="middle">(ViT-L/14 or ResNet-50)</text>

  <!-- Image patches -->
  <rect x="75" y="132" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="103" y="132" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="131" y="132" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="159" y="132" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="75" y="160" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="103" y="160" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="131" y="160" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="159" y="160" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="75" y="188" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="103" y="188" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="131" y="188" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <rect x="159" y="188" width="24" height="24" rx="2" fill="#1e3a5f" stroke="#3b82c4" stroke-width="0.4"/>
  <text x="135" y="228" fill="#8b98a5" font-size="7.5" text-anchor="middle">224×224 → 16×16 patches</text>

  <!-- Projection to embedding -->
  <path d="M135,245 L135,255" fill="none" stroke="#60a5fa" stroke-width="1"/>
  <polygon points="132,253 135,259 138,253" fill="#60a5fa"/>
  <rect x="90" y="260" width="90" height="18" rx="3" fill="#1e3a5f" stroke="#60a5fa" stroke-width="0.6"/>
  <text x="135" y="273" fill="#93c5fd" font-size="7.5" text-anchor="middle">img embedding</text>

  <!-- Text encoder (left side, below image) -->
  <rect x="260" y="85" width="160" height="190" rx="6" fill="#0b1220" stroke="#34d399" stroke-width="1.2"/>
  <text x="340" y="105" fill="#6ee7b7" font-size="10" font-weight="600" text-anchor="middle">Text Encoder</text>
  <text x="340" y="120" fill="#6b7684" font-size="8" text-anchor="middle">(Transformer, 12 layers)</text>

  <!-- Token sequence -->
  <rect x="275" y="138" width="30" height="18" rx="2" fill="#14261f" stroke="#2d6b55" stroke-width="0.4"/>
  <text x="290" y="150" fill="#34d399" font-size="6" text-anchor="middle">[SOS]</text>
  <rect x="309" y="138" width="22" height="18" rx="2" fill="#14261f" stroke="#2d6b55" stroke-width="0.4"/>
  <text x="320" y="150" fill="#34d399" font-size="6" text-anchor="middle">a</text>
  <rect x="335" y="138" width="30" height="18" rx="2" fill="#14261f" stroke="#2d6b55" stroke-width="0.4"/>
  <text x="350" y="150" fill="#34d399" font-size="6" text-anchor="middle">dog</text>
  <rect x="369" y="138" width="30" height="18" rx="2" fill="#14261f" stroke="#2d6b55" stroke-width="0.4"/>
  <text x="384" y="150" fill="#34d399" font-size="6" text-anchor="middle">[EOS]</text>

  <!-- Transformer layers -->
  <rect x="278" y="168" width="120" height="70" rx="4" fill="#0a1a14" stroke="#1a3d2e" stroke-width="0.5"/>
  <text x="338" y="188" fill="#8b98a5" font-size="8" text-anchor="middle">self-attention ×12</text>
  <text x="338" y="204" fill="#8b98a5" font-size="8" text-anchor="middle">causal mask</text>
  <text x="338" y="220" fill="#8b98a5" font-size="8" text-anchor="middle">63M params</text>

  <!-- Projection -->
  <path d="M340,245 L340,255" fill="none" stroke="#34d399" stroke-width="1"/>
  <polygon points="337,253 340,259 343,253" fill="#34d399"/>
  <rect x="295" y="260" width="90" height="18" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.6"/>
  <text x="340" y="273" fill="#6ee7b7" font-size="7.5" text-anchor="middle">text embedding</text>


  <!-- Contrastive matrix (right side) -->
  <rect x="460" y="85" width="255" height="190" rx="6" fill="#0b1220" stroke="#f59e0b" stroke-width="1.2"/>
  <text x="587" y="105" fill="#fbbf24" font-size="10" font-weight="600" text-anchor="middle">Contrastive Matrix (N×N)</text>
  <text x="587" y="120" fill="#6b7684" font-size="8" text-anchor="middle">cosine similarity: img_i · text_j</text>

  <!-- Grid cells -->
  <text x="502" y="140" fill="#8b98a5" font-size="7">I₁</text>
  <text x="502" y="158" fill="#8b98a5" font-size="7">I₂</text>
  <text x="502" y="176" fill="#8b98a5" font-size="7">I₃</text>
  <text x="502" y="194" fill="#8b98a5" font-size="7">I₄</text>

  <text x="530" y="130" fill="#8b98a5" font-size="7" text-anchor="middle">T₁</text>
  <text x="564" y="130" fill="#8b98a5" font-size="7" text-anchor="middle">T₂</text>
  <text x="598" y="130" fill="#8b98a5" font-size="7" text-anchor="middle">T₃</text>
  <text x="632" y="130" fill="#8b98a5" font-size="7" text-anchor="middle">T₄</text>

  <!-- Diagonal = positive pairs (green) -->
  <rect x="517" y="134" width="26" height="14" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
  <text x="530" y="144" fill="#34d399" font-size="7" text-anchor="middle">0.92</text>
  <rect x="551" y="152" width="26" height="14" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
  <text x="564" y="162" fill="#34d399" font-size="7" text-anchor="middle">0.88</text>
  <rect x="585" y="170" width="26" height="14" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
  <text x="598" y="180" fill="#34d399" font-size="7" text-anchor="middle">0.91</text>
  <rect x="619" y="188" width="26" height="14" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
  <text x="632" y="198" fill="#34d399" font-size="7" text-anchor="middle">0.85</text>

  <!-- Off-diagonal = negative pairs (dim) -->
  <rect x="551" y="134" width="26" height="14" rx="2" fill="#1a0a0a" stroke="#475569" stroke-width="0.4"/>
  <text x="564" y="144" fill="#6b7684" font-size="7" text-anchor="middle">0.12</text>
  <rect x="585" y="134" width="26" height="14" rx="2" fill="#1a0a0a" stroke="#475569" stroke-width="0.4"/>
  <text x="598" y="144" fill="#6b7684" font-size="7" text-anchor="middle">0.05</text>
  <rect x="619" y="134" width="26" height="14" rx="2" fill="#1a0a0a" stroke="#475569" stroke-width="0.4"/>
  <text x="632" y="144" fill="#6b7684" font-size="7" text-anchor="middle">0.08</text>

  <rect x="517" y="152" width="26" height="14" rx="2" fill="#1a0a0a" stroke="#475569" stroke-width="0.4"/>
  <text x="530" y="162" fill="#6b7684" font-size="7" text-anchor="middle">0.10</text>
  <rect x="517" y="170" width="26" height="14" rx="2" fill="#1a0a0a" stroke="#475569" stroke-width="0.4"/>
  <text x="530" y="180" fill="#6b7684" font-size="7" text-anchor="middle">0.03</text>
  <rect x="517" y="188" width="26" height="14" rx="2" fill="#1a0a0a" stroke="#475569" stroke-width="0.4"/>
  <text x="530" y="198" fill="#6b7684" font-size="7" text-anchor="middle">0.07</text>

  <text x="587" y="220" fill="#34d399" font-size="8" text-anchor="middle">maximize diagonal (matched pairs)</text>
  <text x="587" y="236" fill="#f87171" font-size="8" text-anchor="middle">minimize off-diagonal (negatives)</text>
  <text x="587" y="255" fill="#6b7684" font-size="7.5" text-anchor="middle">trained on 400M image-text pairs (WIT)</text>
  <text x="587" y="268" fill="#6b7684" font-size="7.5" text-anchor="middle">batch size 32K for hard negatives</text>

  <!-- Bottom: Applications -->
  <rect x="30" y="305" width="700" height="125" rx="6" fill="#080d14" stroke="#233043" stroke-width="1"/>
  <text x="380" y="325" fill="#e6edf3" font-size="10" font-weight="600" text-anchor="middle">CLIP Applications (zero-shot transfer)</text>

  <rect x="55" y="340" width="130" height="55" rx="4" fill="#0b1220" stroke="#60a5fa" stroke-width="0.8"/>
  <text x="120" y="358" fill="#93c5fd" font-size="8.5" font-weight="600" text-anchor="middle">Zero-Shot Classification</text>
  <text x="120" y="374" fill="#8b98a5" font-size="7.5" text-anchor="middle">no training on target dataset</text>
  <text x="120" y="388" fill="#6b7684" font-size="7" text-anchor="middle">"a photo of a {class}"</text>

  <rect x="200" y="340" width="130" height="55" rx="4" fill="#0b1220" stroke="#34d399" stroke-width="0.8"/>
  <text x="265" y="358" fill="#6ee7b7" font-size="8.5" font-weight="600" text-anchor="middle">Image Retrieval</text>
  <text x="265" y="374" fill="#8b98a5" font-size="7.5" text-anchor="middle">text query → nearest images</text>
  <text x="265" y="388" fill="#6b7684" font-size="7" text-anchor="middle">cosine similarity search</text>

  <rect x="345" y="340" width="130" height="55" rx="4" fill="#0b1220" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="410" y="358" fill="#c4b5fd" font-size="8.5" font-weight="600" text-anchor="middle">Image Generation</text>
  <text x="410" y="374" fill="#8b98a5" font-size="7.5" text-anchor="middle">CLIP guides diffusion</text>
  <text x="410" y="388" fill="#6b7684" font-size="7" text-anchor="middle">DALL·E 2, Stable Diffusion</text>

  <rect x="490" y="340" width="130" height="55" rx="4" fill="#0b1220" stroke="#f59e0b" stroke-width="0.8"/>
  <text x="555" y="358" fill="#fbbf24" font-size="8.5" font-weight="600" text-anchor="middle">VLM Backbone</text>
  <text x="555" y="374" fill="#8b98a5" font-size="7.5" text-anchor="middle">vision encoder for LLMs</text>
  <text x="555" y="388" fill="#6b7684" font-size="7" text-anchor="middle">LLaVA, GPT-4V, Gemini</text>

  <text x="380" y="418" fill="#8b98a5" font-size="8.5" text-anchor="middle">CLIP achieves 76.2% zero-shot on ImageNet — competitive with supervised ResNet-50 without seeing any ImageNet labels</text>

  <text x="380" y="452" fill="#6b7684" font-size="11" text-anchor="middle">CLIP proved that natural language supervision scales better than hand-labeled datasets for visual understanding.</text>
</svg>

• Definition: A vision-language model developed by OpenAI that jointly trains an image encoder and a text encoder to produce embeddings in a shared 512-dimensional vector space — images and their matching text descriptions are mapped to nearby points, while non-matching pairs are pushed apart using contrastive loss (InfoNCE).
• Dual Encoder Architecture: The image encoder (Vision Transformer ViT-B/32, ViT-L/14, or ResNet variants) processes images into embedding vectors — the text encoder (12-layer Transformer) processes text into embedding vectors in the same space. Similarity is computed as cosine distance between embeddings.
• Contrastive Training: Given a batch of N image-text pairs, CLIP maximizes the cosine similarity of the N correct pairs while minimizing similarity of the N²-N incorrect pairs — learning to match images with their descriptions rather than predicting fixed class labels.
• Web-Scale Data: Trained on 400M image-text pairs collected from the internet (WebImageText dataset) — the scale and diversity of web data enables CLIP to learn robust visual concepts that transfer across domains without fine-tuning.

Why CLIP Matters

• Zero-Shot Classification: CLIP classifies images into arbitrary categories without training examples — encode class names as text prompts ("a photo of a dog"), compute similarity with the image embedding, and select the highest-scoring class. Achieves competitive accuracy with supervised models on many benchmarks.
• Foundation for Generative AI: CLIP embeddings guide text-to-image generation in Stable Diffusion, DALL-E, and other diffusion models — the text encoder provides the conditioning signal that steers image generation toward the text prompt.
• Universal Image Search: CLIP enables searching image databases using natural language queries — encode the query as text, find images with the most similar embeddings, enabling semantic search that understands concepts rather than just matching keywords.
• Prompt Engineering: Classification accuracy depends on prompt format — "a photo of a {class}" works better than just "{class}" because it matches the distribution of web captions CLIP was trained on.

CLIP Applications

• Image Classification: Zero-shot classification on ImageNet, CIFAR, and domain-specific datasets without fine-tuning.
• Image Search: Natural language search over image databases — "sunset over mountains" finds relevant images by embedding similarity.
• Content Moderation: Detect inappropriate content by computing similarity with text descriptions of policy violations.
• Product Matching: Match product photos to catalog descriptions in e-commerce applications.
• Accessibility: Generate image descriptions for visually impaired users by finding the most similar text descriptions.

CLIP is the foundational vision-language model that revolutionized multimodal AI — demonstrating that contrastive learning on web-scale image-text data enables robust zero-shot visual understanding, powering image search, content moderation, and serving as the text encoder backbone for modern text-to-image generation systems.