Zero-Shot Learning Semantic Embeddings Knowledge Transfer
# Zero-Shot Learning: Semantic Embeddings & Knowledge Transfer
## Introduction & Motivation
Zero-shot learning: classify novel classes without examples; leverage semantic knowledge. Semantic embeddings: map classes to attribute/text space. Knowledge graphs: explicit class relationships. Attribute-based: classes defined by binary attributes. Word embeddings: pretrained semantic vectors. Applications: novel object recognition, taxonomy alignment, knowledge transfer.
Motivation: Real-world: many classes never seen during training. Zero-shot enables classification via semantic relationships.
Applications: Image classification (new categories), machine translation (new language pairs), product recommendation (new items).
---
## Core Concepts & Theory
### Semantic Embeddings
Classes represented as vectors in semantic space (text, attributes).
### Attribute-Based Classification
Classes defined by binary attributes; classify via attribute prediction.
### Word Embeddings
Pretrained vectors (GloVe, Word2Vec); leverage linguistic knowledge.
---
## Mathematical Formulation
Attribute-based classification:
$$P(y = c | x) \propto ext{similarity}(f(x), ext{attributes}_c)$$
where attributes_c = binary vector for class c.
Word embedding classification:
$$P(y = c | x) = ext{softmax}( ext{sim}(f(x), ext{embed}_c))$$
embed_c = pretrained word vector for class name.
Domain alignment:
$$L = L_{ ext{seen}} + \lambda \|f(x_{ ext{unseen}}) - g( ext{embed}_{ ext{unseen}})\|^2$$
align visual and semantic spaces.
---
## Advanced Theory & Extensions
### Generalized Zero-Shot Learning
Test on both seen and unseen classes; avoid bias to seen.
### Transductive Zero-Shot Learning
Use unlabeled target examples; semi-supervised learning.
### Cross-Lingual Transfer
Zero-shot via language embeddings; translate between languages.
---
## Computational Considerations
Semantic embedding: O(1) lookup per class.
Similarity computation: O(C) for C classes.
Embedding alignment: O(N·D) for N samples, D dimensions.
---
## Practical Implementation Strategies
### Semantic Source
Use pretrained embeddings (GloVe, Word2Vec, fastText).
### Attribute Engineering
Manual attribute definition or learned via auxiliary data.
### Domain Alignment
Train joint embedding space; visual and semantic.
---
## Benchmark Datasets & Evaluation
AWA (Animals): Attribute-based; ~50 animals, 85 attributes.
CUB (Birds): Fine-grained; ~200 species, semantic attributes.
ImageNet: Zero-shot on held-out synsets; 21841 classes.
---
## Key Challenges & Limitations
### Semantic Gap
Visual-semantic mismatch; harder classes to distinguish.
### Attribute Bias
Some attributes easier to predict; bias in predictions.
### Seen-Unseen Bias
Test set bias to seen classes; generalized setting harder.
---
## Hyperparameter Tuning
Embedding dimension: 100-300 typical; balance expressiveness-efficiency.
Alignment weight λ: 0.1-1.0; balance reconstruction-alignment.
Margin (contrastive): 0.1-1.0; control decision boundary.
---
## Real-World Applications & Case Studies
Image Tagging: New tags via semantic embeddings; no training.
Product Recommendation: New products via category embeddings.
Machine Translation: Unseen language pairs via pivot languages.
---
## Integration with Other Methods
Zero-Shot + Few-Shot → continual learning.
Zero-Shot + Generative Models → synthetic data for unseen.
---
## Summary & Key Takeaways
Zero-shot learning via semantic embeddings and attribute-based classification enables novel class recognition through knowledge transfer from semantic representations.
Principles:
1. Semantic embeddings: classes as vectors in attribute/text space.
2. Attribute-based: explicit class definitions.
3. Word embeddings: leverage linguistic knowledge.
4. Domain alignment: joint visual-semantic space.
5. Generalized setting: both seen and unseen at test.
---
---
## Appendix: Practical Labs
### Lab 1: Semantic Embeddings
import numpy as np
def create_semantic_embeddings(class_names, embedding_dim=100):
"""Create semantic embeddings for classes (simulated)"""
embeddings = {}
np.random.seed(42)
for c in class_names:
# Simulate pretrained embedding (e.g., GloVe)
emb = np.random.randn(embedding_dim)
emb = emb / np.linalg.norm(emb) # Normalize
embeddings[c] = emb
return embeddings
def zero_shot_classify(x_feature, class_embeddings):
"""Classify via semantic similarity"""
scores = {}
for class_name, class_emb in class_embeddings.items():
# Cosine similarity
score = np.dot(x_feature, class_emb) / (np.linalg.norm(x_feature) * np.linalg.norm(class_emb) + 1e-8)
scores[class_name] = score
predicted_class = max(scores, key=scores.get)
return predicted_class, scores
# Test
class_names = ['dog', 'cat', 'bird', 'fish']
embeddings = create_semantic_embeddings(class_names, embedding_dim=100)
x_feature = np.random.randn(100)
x_feature = x_feature / np.linalg.norm(x_feature)
predicted, scores = zero_shot_classify(x_feature, embeddings)
assert predicted in class_names, "Predicted class should be valid"
assert len(scores) == 4, "Should have scores for all classes"
print("✓ Zero-shot classification working")
if __name__ == "__main__":
print("Lab 1: ZeroShot - PASSED")### Lab 2: Attribute-Based Classification
import numpy as np
class AttributeClassifier:
def __init__(self, attribute_matrix):
"""attribute_matrix: [n_classes, n_attributes]"""
self.attributes = attribute_matrix
def classify(self, predicted_attributes, threshold=0.5):
"""Classify based on predicted attributes"""
# Binarize predictions
pred_binary = (predicted_attributes > threshold).astype(int)
# Compute similarity to each class
similarities = []
for class_attr in self.attributes:
sim = np.sum(pred_binary == class_attr) / len(class_attr)
similarities.append(sim)
predicted_class = np.argmax(similarities)
return predicted_class, similarities
# Test
np.random.seed(42)
n_classes = 5
n_attributes = 10
# Create attribute matrix
attribute_matrix = np.random.randint(0, 2, (n_classes, n_attributes))
classifier = AttributeClassifier(attribute_matrix)
# Test prediction
predicted_attributes = np.random.uniform(0, 1, n_attributes)
pred_class, sims = classifier.classify(predicted_attributes)
assert pred_class in range(n_classes), "Predicted class valid"
assert len(sims) == n_classes, "Should have similarity per class"
print("✓ Attribute classification working")
if __name__ == "__main__":
print("Lab 2: Attributes - PASSED")### Lab 3: Visual-Semantic Alignment
import numpy as np
def align_visual_semantic(visual_features, semantic_embeddings, labels, learning_rate=0.01, iterations=10):
"""Learn alignment between visual and semantic spaces"""
n_samples, visual_dim = visual_features.shape
semantic_dim = semantic_embeddings[0].shape[0]
# Projection matrix: visual -> semantic
W = np.eye(semantic_dim, visual_dim) * np.sqrt(semantic_dim / visual_dim)
losses = []
for iteration in range(iterations):
loss = 0
dW = np.zeros_like(W)
for i in range(n_samples):
visual = visual_features[i]
semantic = semantic_embeddings[labels[i]]
# Project visual to semantic space
projected = W @ visual
# Alignment loss
diff = projected - semantic
loss += np.sum(diff ** 2)
# Gradient
dW += 2 * np.outer(diff, visual)
# Update W
W = W - learning_rate * (dW / n_samples)
losses.append(loss / n_samples)
return W, losses
# Test
np.random.seed(42)
visual_features = np.random.randn(20, 64)
semantic_embeddings = [np.random.randn(32) for _ in range(5)]
labels = np.random.randint(0, 5, 20)
W, losses = align_visual_semantic(visual_features, semantic_embeddings, labels)
assert W.shape == (32, 64), "Projection matrix shape correct"
assert losses[-1] < losses[0], "Loss should decrease"
print("✓ Visual-semantic alignment working")
if __name__ == "__main__":
print("Lab 3: Alignment - PASSED")### Lab 4: Generalized Zero-Shot Learning
import numpy as np
def compute_gzsl_metrics(y_true, y_pred, seen_classes, unseen_classes):
"""Compute generalized zero-shot metrics"""
# Accuracy on seen classes
seen_mask = np.isin(y_true, seen_classes)
seen_acc = np.mean(y_true[seen_mask] == y_pred[seen_mask])
# Accuracy on unseen classes
unseen_mask = np.isin(y_true, unseen_classes)
unseen_acc = np.mean(y_true[unseen_mask] == y_pred[unseen_mask])
# Harmonic mean (H-score)
h_score = 2 * (seen_acc * unseen_acc) / (seen_acc + unseen_acc + 1e-8)
return seen_acc, unseen_acc, h_score
# Test
np.random.seed(42)
n_samples = 100
seen_classes = [0, 1, 2]
unseen_classes = [3, 4]
y_true = np.random.choice(range(5), n_samples)
y_pred = np.random.choice(range(5), n_samples)
seen_acc, unseen_acc, h_score = compute_gzsl_metrics(y_true, y_pred, seen_classes, unseen_classes)
assert 0 <= seen_acc <= 1, "Seen accuracy in [0,1]"
assert 0 <= unseen_acc <= 1, "Unseen accuracy in [0,1]"
assert 0 <= h_score <= 1, "H-score in [0,1]"
print("✓ Generalized ZSL metrics working")
if __name__ == "__main__":
print("Lab 4: GZSL - PASSED")