Clip - Vision-Language Pre-Training

# CLIP - Vision-Language Pre-Training

## Introduction & Motivation

CLIP: contrastive learning of image-text pairs. Multimodal representation learning. Applications: zero-shot image classification, image-text retrieval.

Motivation: Learn joint image-text representations.

Applications: Zero-shot classification, cross-modal retrieval.

---

## Core Concepts & Theory

### Image Encoder

Visual feature extraction.

### Text Encoder

Linguistic feature extraction.

### Contrastive Loss

Alignment of modalities.

### Zero-Shot Transfer

Classification without fine-tuning.

---

## Mathematical Formulation

Contrastive Loss:
$$\mathcal{L} = -\log \frac{\exp( ext{sim}(I_i, T_i) / au)}{\sum_j \exp( ext{sim}(I_i, T_j) / au)}$$

Cosine Similarity:
$$ ext{sim}(I, T) = \frac{I \cdot T}{||I|| ||T||}$$

Zero-Shot Classification:
$$p(c|I) = \frac{\exp( ext{sim}(I, T_c))}{\sum_k \exp( ext{sim}(I, T_k))}$$

---

## Advanced Theory & Extensions

### Multi-Scale Resolution

Handling variable image sizes.

### Domain-Specific Fine-Tuning

Task adaptation.

### Efficient CLIP Variants

Reduced model sizes.

---

## Computational Considerations

Image encoding: O(H·W·C·K²).

Text encoding: O(T·D).

Contrastive loss: O(B²·D).

---

## Practical Implementation Strategies

### Synthetic Captions

Text augmentation.

### Hard Negative Mining

Difficult sample selection.

### Gradient Accumulation

Large batch simulation.

---

## Benchmark Datasets & Evaluation

ImageNet: Zero-shot classification.

Flickr30K: Image-text retrieval.

COCO: Dense captioning.

---

## Key Challenges & Limitations

### Language Bias

Text biases in data.

### Domain Shift

Generalization to new domains.

### Computational Cost

Large-scale training expensive.

---

## Hyperparameter Tuning

Temperature: 0.05-0.1.

Learning rate: 1e-3 to 5e-3.

Batch size: 256-1024.

---

## Real-World Applications & Case Studies

Zero-Shot Classification: New category prediction.

Image Retrieval: Cross-modal search.

Accessibility: Image description generation.

---

## Integration with Other Methods

CLIP + task-specific adapters; + text prompts for classification.

---

## Summary & Key Takeaways

CLIP learns joint image-text representations via contrastive learning.

Principles:
1. Contrastive learning: Modality alignment.
2. Image encoding: Visual feature extraction.
3. Text encoding: Linguistic representation.
4. Zero-shot: Task generalization.
5. Multimodal: Cross-modal understanding.

---

## Appendix: Practical Labs

### Lab 1: Image-Text Matching

import numpy as np

def compute_image_text_similarity(image_features, text_features):
 """Compute similarity between images and texts"""
 similarities = image_features @ text_features.T
 return similarities

np.random.seed(42)
img_feat = np.random.randn(32, 512)
img_feat /= np.linalg.norm(img_feat, axis=1, keepdims=True)
txt_feat = np.random.randn(10, 512)
txt_feat /= np.linalg.norm(txt_feat, axis=1, keepdims=True)
sims = compute_image_text_similarity(img_feat, txt_feat)
assert sims.shape == (32, 10), "Correct similarity shape"
print("✓ Image-text similarity working")

### Lab 2: Contrastive Loss

import numpy as np

def clip_contrastive_loss(image_features, text_features, temperature=0.07):
 """Compute CLIP contrastive loss"""
 batch_size = image_features.shape[0]
 
 # Compute similarities
 logits = (image_features @ text_features.T) / temperature
 
 # Cross-entropy with diagonal as targets
 loss = 0.0
 for i in range(batch_size):
 probs = np.exp(logits[i]) / np.sum(np.exp(logits[i]))
 loss -= np.log(probs[i] + 1e-8)
 
 return loss / batch_size

np.random.seed(42)
img = np.random.randn(8, 512)
txt = np.random.randn(8, 512)
img /= np.linalg.norm(img, axis=1, keepdims=True)
txt /= np.linalg.norm(txt, axis=1, keepdims=True)
loss = clip_contrastive_loss(img, txt)
assert loss > 0, "Positive loss"
print(f"✓ CLIP loss working: {loss:.4f}")

### Lab 3: Zero-Shot Prediction

import numpy as np

def zero_shot_classification(image_features, class_text_features):
 """Predict class from image using text embeddings"""
 similarities = image_features @ class_text_features.T
 predictions = np.argmax(similarities, axis=1)
 return predictions

np.random.seed(42)
img_feat = np.random.randn(10, 512)
img_feat /= np.linalg.norm(img_feat, axis=1, keepdims=True)
class_feat = np.random.randn(10, 512) # 10 classes
class_feat /= np.linalg.norm(class_feat, axis=1, keepdims=True)
preds = zero_shot_classification(img_feat, class_feat)
assert preds.shape == (10,), "Correct prediction shape"
print("✓ Zero-shot classification working")

### Lab 4: Text Template Generation

def generate_text_templates(class_name):
 """Generate text templates for zero-shot classification"""
 templates = [
 f"a photo of {class_name}",
 f"a photo of a {class_name}",
 f"a picture of {class_name}",
 f"{class_name}",
 f"the {class_name}"
 ]
 return templates

templates = generate_text_templates("cat")
assert len(templates) > 0, "Templates generated"
print(f"✓ Text templates: {templates[0]}")

---