Few-Shot Learning Prototypical Networks Support-Query Framework

# Few-Shot Learning: Prototypical Networks & Support-Query Framework

## Introduction & Motivation

Few-shot learning learns from few labeled examples per class (k-shot, n-way). Prototypical networks: learn metric space; class prototype = mean support embeddings; classify by distance to prototypes. Meta-learning: learn to learn. Critical for rapid adaptation, data-limited domains.

Motivation: Deep learning requires large labeled sets. Few-shot learns from minimal data; generalizes to new tasks.

Applications: One-shot face recognition, medical diagnosis, rare disease detection, low-resource languages.

---

## Core Concepts & Theory

### Prototypical Networks

Per-class embedding prototype: c_k = mean_i φ(x_i) for x_i in class k.
Query classified to nearest prototype in embedding space.

### Support-Query Split

Support set: few labeled examples. Query set: test examples.

### Metric Learning

Learn distance metric (Euclidean, Cosine) to minimize classification error.

---

## Mathematical Formulation

Prototype:
$$c_k = \frac{1}{|S_k|} \sum_{(x,y) \in S_k} \phi_ heta(x)$$

Distance-based classification:
$$\hat{y}_q = \arg\min_k d(\phi_ heta(x_q), c_k)$$

Prototypical loss (cross-entropy in metric space):
$$\mathcal{L} = -\sum_q \log \frac{\exp(-d(f_q, c_{y_q}))}{\sum_k \exp(-d(f_q, c_k))}$$

---

## Advanced Theory & Extensions

### Matching Networks

Attention + embedding; learn task-conditioned classifier.

### Relation Networks

Learn similarity via siamese network; more flexible than fixed metric.

### Meta-Learning (MAML)

Learn initialization for rapid gradient-based adaptation.

---

## Computational Considerations

Per-task training: O(support_size × embedding_dim).

Prototype computation: O(k × n × d) for k classes, n shots.

Distance queries: O(k) per query sample.

---

## Practical Implementation Strategies

### Data Augmentation

Augment support set; improve prototype robustness.

### Episodic Training

Train on few-shot tasks sampled from dataset.

### Transductive Inference

Use unlabeled query set to refine prototypes.

---

## Benchmark Datasets & Evaluation

mini-ImageNet: 100 classes, 600 images/class; 5-way, 5-shot standard.

Omniglot: 1623 classes, 20 images/class; rapid learning.

Metrics: Accuracy on test tasks; convergence speed.

---

## Key Challenges & Limitations

### Domain Shift

Prototypes may not transfer to different domains.

### Limited Support

Prototype estimation noisy with few examples.

### Class Imbalance

Few-shot setup naturally imbalanced.

---

## Hyperparameter Tuning

Embedding dimension: 64-512.

Distance metric: Euclidean, Cosine.

Ways & Shots: 5-way, 5-shot standard; vary for challenge.

---

## Real-World Applications & Case Studies

Face Recognition: One-shot face identification.

Medical Imaging: Learn disease patterns from few examples.

NLP: Few-shot text classification via prototypes.

---

## Integration with Other Methods

Few-Shot + Uncertainty → Bayesian prototypes.

Few-Shot + Transfer → pretrain, finetune on target task.

---

## Summary & Key Takeaways

Few-shot learning leverages prototypical networks to classify via distance to class prototypes learned from minimal examples.

Principles:
1. Prototype = mean support embeddings.
2. Query classified to nearest prototype.
3. Episodic training matches test task structure.
4. Metric learning improves embedding quality.
5. Transductive inference leverages query set.

---

---

## Appendix: Practical Labs

### Lab 1: Prototypical Networks

import torch
import torch.nn as nn
import numpy as np

class PrototypicalNet(nn.Module):
 def __init__(self, input_dim=28*28, embedding_dim=64):
 super().__init__()
 self.embedding = nn.Sequential(
 nn.Linear(input_dim, 128),
 nn.ReLU(),
 nn.Linear(128, embedding_dim)
 )
 
 def forward(self, x):
 return self.embedding(x)

# Data
np.random.seed(42)
n_ways, n_shots, n_queries = 3, 5, 10
support_x = torch.randn(n_ways * n_shots, 28*28)
support_y = torch.repeat_interleave(torch.arange(n_ways), n_shots)
query_x = torch.randn(n_ways * n_queries, 28*28)
query_y = torch.repeat_interleave(torch.arange(n_ways), n_queries)

model = PrototypicalNet(input_dim=28*28, embedding_dim=64)

# Compute prototypes
embeddings_support = model(support_x)
prototypes = torch.stack([
 embeddings_support[support_y == k].mean(dim=0) for k in range(n_ways)
])

# Classify queries
embeddings_query = model(query_x)
distances = torch.cdist(embeddings_query, prototypes)
predictions = torch.argmin(distances, dim=1)

accuracy = (predictions == query_y).float().mean()
print(f"Prototypical Networks accuracy: {accuracy:.2%}")
assert 0 <= accuracy <= 1, "Accuracy should be in [0,1]"
print("✓ Prototypical networks working")

if __name__ == "__main__":
 print("Lab 1: Prototypical - PASSED")

### Lab 2: Support-Query Episodic Training

import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np

class SimpleEmbedder(nn.Module):
 def __init__(self):
 super().__init__()
 self.net = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 16))
 
 def forward(self, x):
 return self.net(x)

def episodic_training_step(model, support_x, support_y, query_x, query_y, optimizer, n_ways):
 optimizer.zero_grad()
 
 # Embeddings
 emb_support = model(support_x)
 emb_query = model(query_x)
 
 # Prototypes
 prototypes = torch.stack([emb_support[support_y == k].mean(dim=0) for k in range(n_ways)])
 
 # Distances
 distances = torch.cdist(emb_query, prototypes)
 logits = -distances
 
 loss = nn.functional.cross_entropy(logits, query_y)
 loss.backward()
 optimizer.step()
 
 return loss.item()

model = SimpleEmbedder()
optimizer = optim.Adam(model.parameters(), lr=0.01)

losses = []
for episode in range(20):
 support_x = torch.randn(15, 10)
 support_y = torch.tensor([0]*5 + [1]*5 + [2]*5)
 query_x = torch.randn(9, 10)
 query_y = torch.tensor([0]*3 + [1]*3 + [2]*3)
 
 loss = episodic_training_step(model, support_x, support_y, query_x, query_y, optimizer, n_ways=3)
 losses.append(loss)

print(f"Final episodic loss: {losses[-1]:.4f}")
assert len(losses) == 20, "Should have 20 episodes"
assert all(np.isfinite(l) for l in losses), "All losses should be finite"
print("✓ Episodic training working")

if __name__ == "__main__":
 print("Lab 2: Episodic - PASSED")

### Lab 3: Metric Learning

import torch
import torch.nn as nn
import numpy as np

class MetricLearner(nn.Module):
 def __init__(self, input_dim=10, embedding_dim=16):
 super().__init__()
 self.embedding = nn.Linear(input_dim, embedding_dim)
 
 def forward(self, x):
 return self.embedding(x)
 
 def compute_distances(self, x1, x2):
 emb1 = self.embedding(x1)
 emb2 = self.embedding(x2)
 return torch.cdist(emb1, emb2)

model = MetricLearner(input_dim=10, embedding_dim=16)

# Positive pair (same class)
x1 = torch.randn(1, 10)
x2 = x1 + 0.1 * torch.randn(1, 10)

# Negative pair (different class)
x3 = torch.randn(1, 10)

dist_pos = model.compute_distances(x1, x2)
dist_neg = model.compute_distances(x1, x3)

print(f"Positive distance: {dist_pos.item():.4f}")
print(f"Negative distance: {dist_neg.item():.4f}")
assert dist_pos.item() >= 0, "Distance should be non-negative"
assert dist_neg.item() >= 0, "Distance should be non-negative"
print("✓ Metric learning working")

if __name__ == "__main__":
 print("Lab 3: Metric - PASSED")

### Lab 4: Few-Shot Classification Pipeline

import torch
import torch.nn as nn
import numpy as np

def few_shot_classify(support_x, support_y, query_x, embedding_fn, n_ways):
 """End-to-end few-shot classification"""
 emb_support = embedding_fn(support_x)
 emb_query = embedding_fn(query_x)
 
 # Compute class prototypes
 prototypes = []
 for k in range(n_ways):
 mask = support_y == k
 prototype = emb_support[mask].mean(dim=0)
 prototypes.append(prototype)
 prototypes = torch.stack(prototypes)
 
 # Classify queries
 distances = torch.cdist(emb_query, prototypes)
 predictions = torch.argmin(distances, dim=1)
 
 return predictions

class SimpleEmbedder(nn.Module):
 def __init__(self):
 super().__init__()
 self.net = nn.Sequential(nn.Linear(20, 32), nn.ReLU(), nn.Linear(32, 8))
 def forward(self, x):
 return self.net(x)

np.random.seed(42)
support_x = torch.randn(15, 20)
support_y = torch.tensor([0]*5 + [1]*5 + [2]*5)
query_x = torch.randn(9, 20)
query_y = torch.tensor([0]*3 + [1]*3 + [2]*3)

embedding_fn = SimpleEmbedder()
predictions = few_shot_classify(support_x, support_y, query_x, embedding_fn, n_ways=3)

accuracy = (predictions == query_y).float().mean()
print(f"Few-shot accuracy: {accuracy:.2%}")
assert 0 <= accuracy <= 1, "Accuracy in [0,1]"
print("✓ Few-shot pipeline working")

if __name__ == "__main__":
 print("Lab 4: Pipeline - PASSED")