Flamingo - Multimodal In-Context Learning

# Flamingo - Multimodal In-Context Learning

## Introduction & Motivation

Flamingo: multimodal model with in-context learning. Visual and textual understanding. Applications: image-text tasks, reasoning about images.

Motivation: Enable multimodal in-context learning at scale.

Applications: Visual QA, image captioning, visual reasoning.

---

## Core Concepts & Theory

### Visual Tokens

Image patch embeddings.

### Gated Cross-Attention

Fuse vision and language.

### In-Context Examples

Learn from demonstrations.

### Freezing Backbone

Efficient fine-tuning.

---

## Mathematical Formulation

Visual Encoding:
$$V = ext{VisualEncoder}(I)$$

Gated Fusion:
$$h_t = ext{GatedCA}(h_t, V)$$

Language Generation:
$$P(w_t) = ext{softmax}(W h_t)$$

---

## Advanced Theory & Extensions

### Efficient Adaptation

LoRA for parameters.

### Cross-Modal Reasoning

Relate vision and language.

### Few-Shot Transfer

Rapid task adaptation.

---

## Computational Considerations

Visual encoding: O(patches·D).

Gated fusion: O(T·D²).

Generation: Standard LLM.

---

## Practical Implementation Strategies

### Prompt Structure

Mix images and text.

### Example Selection

Choose representative samples.

### Fine-tuning

Adapt for specific tasks.

---

## Benchmark Datasets & Evaluation

Visual QA: VQA v2.

Image Captioning: COCO.

Visual Reasoning: Reasoning tasks.

---

## Key Challenges & Limitations

### Alignment

Vision-language mismatch.

### Efficiency

Multimodal processing cost.

### Scalability

Training data requirements.

---

## Hyperparameter Tuning

Visual dimension: 512-1024.

Language dimension: 512-2048.

Learning rate: 1e-5 to 1e-4.

---

## Real-World Applications & Case Studies

Image QA: Answer visual questions.

Captioning: Generate descriptions.

Reasoning: Visual reasoning tasks.

---

## Integration with Other Methods

Flamingo + vision models; + language models.

---

## Summary & Key Takeaways

Flamingo enables multimodal in-context learning.

Principles:
1. Multimodal: Vision and language.
2. In-context: Few-shot adaptation.
3. Gated fusion: Efficient combination.
4. Freezing: Efficient training.
5. Scalability: Works across tasks.

---

## Appendix: Practical Labs

### Lab 1: Gated Cross-Attention

import numpy as np

def gated_cross_attention(language_hidden, visual_features, gate_param=0.5):
 """Fuse language and visual via gating"""
 # Cross-attention
 attn = language_hidden @ visual_features.T
 attn = np.exp(attn) / np.sum(np.exp(attn), axis=-1, keepdims=True)
 visual_context = attn @ visual_features
 
 # Gating
 gated = gate_param * language_hidden + (1 - gate_param) * visual_context
 return gated

np.random.seed(42)
lang = np.random.randn(10, 256)
vis = np.random.randn(49, 256)
out = gated_cross_attention(lang, vis)
assert out.shape == lang.shape
print("✓ Gated cross-attention working")

### Lab 2: Visual Token Extraction

import numpy as np

def extract_visual_tokens(image, patch_size=16):
 """Extract visual tokens from image"""
 h, w = image.shape[:2]
 patches = []
 
 for i in range(0, h, patch_size):
 for j in range(0, w, patch_size):
 patch = image[i:i+patch_size, j:j+patch_size]
 patches.append(patch.flatten())
 
 return np.array(patches)

np.random.seed(42)
img = np.random.rand(224, 224, 3)
tokens = extract_visual_tokens(img, 16)
assert tokens.shape[0] == (224//16) ** 2
print(f"✓ Visual tokens: {tokens.shape}")

### Lab 3: Multimodal Prompt

def construct_multimodal_prompt(images, texts, interleave=True):
 """Build multimodal prompt"""
 if interleave:
 # Interleave images and text
 prompt = []
 for img, txt in zip(images, texts):
 prompt.append(f"<image>{img}</image>")
 prompt.append(f"<text>{txt}</text>")
 else:
 # Sequential
 prompt = [f"<image>{img}</image>" for img in images]
 prompt += [f"<text>{txt}</text>" for txt in texts]
 
 return " ".join(prompt)

prompt = construct_multimodal_prompt(["img1", "img2"], ["caption1", "caption2"])
assert "<image>" in prompt
print("✓ Multimodal prompt created")

### Lab 4: Few-Shot Multimodal Adaptation

import numpy as np

def adapt_multimodal(in_context_examples, query, embed_fn):
 """Adapt model via multimodal examples"""
 # Embed examples
 example_embs = []
 for ex in in_context_examples:
 emb = embed_fn(ex)
 example_embs.append(emb)
 
 # Find similar examples
 query_emb = embed_fn(query)
 similarities = [np.dot(query_emb, e) for e in example_embs]
 best_idx = np.argmax(similarities)
 
 return in_context_examples[best_idx]

examples = ["img1+text1", "img2+text2"]
embed = lambda x: np.random.randn(256)
best = adapt_multimodal(examples, "query", embed)
assert best in examples
print("✓ Multimodal adaptation working")

---