Multimodal Learning Vision-Language Cross-Modal

# Multimodal Learning: Vision-Language & Cross-Modal

## Introduction & Motivation

Multimodal Learning: learn from multiple modalities. Vision-language models (CLIP, BLIP). Cross-modal alignment; fusion. Applications: image captioning, VQA, retrieval.

Motivation: Combine complementary information; richer understanding.

Applications: VQA, captioning, retrieval, alignment.

---

## Core Concepts & Theory

### Alignment Loss

Match image-text representations.

### Contrastive Learning

Cross-modal contrastive pairs.

### Fusion Strategies

Early, late, or hybrid fusion.

---

## Mathematical Formulation

Cross-modal contrastive loss:
$$L = -\log \frac{\exp( ext{sim}(I, T) / au)}{\sum_k \exp( ext{sim}(I, T_k) / au)}$$

Alignment objective:
$$L_{ ext{align}} = \|f_I(I) - f_T(T)\|^2$$

Fusion:
$$z = ext{MLP}([f_I(I); f_T(T)])$$

---

## Advanced Theory & Extensions

### CLIP

Contrastive language-image.

### BLIP

Vision-language pre-training.

### Attention-based Fusion

Cross-modal attention.

---

## Computational Considerations

Vision encoder: O(image_size).

Language encoder: O(text_length).

Alignment: O(batch_size²).

---

## Practical Implementation Strategies

### Pre-training

Large-scale image-text pairs.

### Fine-tuning

Task-specific adaptation.

### Modality Balancing

Equal importance; temperature.

---

## Benchmark Datasets & Evaluation

Conceptual Captions: Large image-text.

COCO Captions: Dense captioning.

Flickr30K: Image-text retrieval.

---

## Key Challenges & Limitations

### Modality Gap

Different feature distributions.

### Alignment Quality

Noisy text descriptions.

### Scalability

Large datasets required.

---

## Hyperparameter Tuning

Temperature τ: 0.07-0.1; confidence.

Fusion ratio: Balanced loss weighting.

Learning rate: 1e-4 to 1e-3.

---

## Real-World Applications & Case Studies

Image Captioning: Automatic descriptions.

VQA: Visual question answering.

Retrieval: Cross-modal search.

---

## Integration with Other Methods

Multimodal + Downstream → task adaptation.

Multimodal + Ensemble → robust fusion.

---

## Summary & Key Takeaways

Multimodal Learning via cross-modal alignment enables rich understanding through vision-language fusion and contrastive learning.

Principles:
1. Modalities: complementary signals.
2. Alignment: semantic matching.
3. Contrastive: negative sampling.
4. Fusion: combine representations.
5. Transfer: pre-trained models.

---

---

## Appendix: Practical Labs

### Lab 1: Image-Text Matching

import numpy as np

def compute_similarity_matrix(image_features, text_features):
 """Compute similarity between images and texts"""
 # Normalize
 img_norm = image_features / (np.linalg.norm(image_features, axis=1, keepdims=True) + 1e-8)
 txt_norm = text_features / (np.linalg.norm(text_features, axis=1, keepdims=True) + 1e-8)
 
 # Cosine similarity
 similarity = img_norm @ txt_norm.T
 
 return similarity

# Test
np.random.seed(42)
img_feat = np.random.randn(32, 256)
txt_feat = np.random.randn(32, 256)

sim = compute_similarity_matrix(img_feat, txt_feat)

assert sim.shape == (32, 32), "Similarity shape"
print("✓ Similarity computation working")

if __name__ == "__main__":
 print("Lab 1: Similarity - PASSED")

### Lab 2: Cross-Modal Contrastive Loss

import numpy as np

def cross_modal_contrastive_loss(image_features, text_features, temperature=0.07):
 """Cross-modal contrastive loss"""
 # Normalize
 img_norm = image_features / (np.linalg.norm(image_features, axis=1, keepdims=True) + 1e-8)
 txt_norm = text_features / (np.linalg.norm(text_features, axis=1, keepdims=True) + 1e-8)
 
 # Similarity matrix
 sim_matrix = img_norm @ txt_norm.T / temperature
 
 # Loss
 batch_size = len(image_features)
 labels = np.arange(batch_size)
 
 exp_sim = np.exp(sim_matrix - np.max(sim_matrix, axis=1, keepdims=True))
 probs = exp_sim / exp_sim.sum(axis=1, keepdims=True)
 
 loss = -np.log(probs[np.arange(batch_size), labels] + 1e-8).mean()
 
 return loss

# Test
np.random.seed(42)
img = np.random.randn(32, 256)
txt = np.random.randn(32, 256)

loss = cross_modal_contrastive_loss(img, txt)

assert np.isfinite(loss), "Loss finite"
print("✓ Cross-modal loss working")

if __name__ == "__main__":
 print("Lab 2: CrossModalLoss - PASSED")

### Lab 3: Multimodal Fusion

import numpy as np

def fuse_modalities(image_features, text_features, fusion_type='concat'):
 """Fuse image and text features"""
 if fusion_type == 'concat':
 fused = np.concatenate([image_features, text_features], axis=1)
 elif fusion_type == 'mean':
 fused = (image_features + text_features) / 2
 elif fusion_type == 'attention':
 # Simple attention-based fusion
 alpha = 0.6
 fused = alpha * image_features + (1 - alpha) * text_features
 else:
 raise ValueError(f"Unknown fusion: {fusion_type}")
 
 return fused

# Test
np.random.seed(42)
img = np.random.randn(32, 256)
txt = np.random.randn(32, 256)

fused_concat = fuse_modalities(img, txt, 'concat')
fused_mean = fuse_modalities(img, txt, 'mean')

assert fused_concat.shape == (32, 512), "Concat shape"
assert fused_mean.shape == (32, 256), "Mean shape"
print("✓ Multimodal fusion working")

if __name__ == "__main__":
 print("Lab 3: MultimodalFusion - PASSED")

### Lab 4: Retrieval Evaluation

import numpy as np

def compute_retrieval_metrics(similarity_matrix, k=5):
 """Evaluate image-text retrieval"""
 # Diagonal has matching pairs
 batch_size = len(similarity_matrix)
 
 # Recall@K
 recalls = []
 for i in range(batch_size):
 # Top-k indices
 top_k = np.argsort(similarity_matrix[i])[-k:]
 
 if i in top_k:
 recalls.append(1)
 else:
 recalls.append(0)
 
 recall_at_k = np.mean(recalls)
 
 return recall_at_k

# Test
np.random.seed(42)
sim = np.random.randn(32, 32)

recall = compute_retrieval_metrics(sim, k=5)

assert 0 <= recall <= 1, "Recall in [0,1]"
print("✓ Retrieval metrics working")

if __name__ == "__main__":
 print("Lab 4: RetrievalMetrics - PASSED")