Sentence Transformers - Semantic Similarity

# Sentence Transformers - Semantic Similarity

## Introduction & Motivation

Sentence Transformers: fine-tune transformers for semantic textual similarity. Siamese networks for sentence embeddings. Applications: semantic search, clustering, recommendation.

Motivation: Generate meaningful sentence embeddings for similarity tasks.

Applications: Semantic search, document clustering, recommendation systems.

---

## Core Concepts & Theory

### Siamese Networks

Identical networks processing paired sentences.

### Triplet Loss

Learn ordering via positive/negative examples.

### Contrastive Loss

Minimize distance to similar, maximize to dissimilar.

### Pooling Strategies

Extract sentence representation from token embeddings.

---

## Mathematical Formulation

Triplet Loss:
$$\mathcal{L} = \max(d(a, p) - d(a, n) + \alpha, 0)$$

Contrastive Loss:
$$\mathcal{L} = -\log \frac{\exp( ext{sim}(s_i, s_j) / au)}{\sum_k \exp( ext{sim}(s_i, s_k) / au)}$$

Mean Pooling:
$$s = \frac{1}{T} \sum_t h_t$$

---

## Advanced Theory & Extensions

### Multiple Pooling Methods

CLS token, mean, max pooling.

### Hard Negative Mining

Focus on challenging examples.

### In-batch Negatives

Efficient negative sampling.

---

## Computational Considerations

Forward pass: O(T·D²).

Pooling: O(T·D).

Similarity computation: O(N·D).

---

## Practical Implementation Strategies

### Pair Selection

Choose positive/negative pairs.

### Data Augmentation

Create similar sentence pairs.

### Warm-up Training

Multi-stage training.

---

## Benchmark Datasets & Evaluation

STS Benchmark: Semantic similarity.

SQuAD: Dense retrieval.

TREC-COVID: Document ranking.

---

## Key Challenges & Limitations

### Computational Cost

Pairwise comparisons expensive.

### Hard Negative Mining

Challenging example selection.

### Domain Specificity

Transfer to new domains difficult.

---

## Hyperparameter Tuning

Margin (alpha): 0.5-2.0.

Temperature: 0.05-0.1.

Learning rate: 1e-5 to 1e-4.

---

## Real-World Applications & Case Studies

Semantic Search: Find similar documents.

Clustering: Group similar sentences.

Recommendation: Suggest related content.

---

## Integration with Other Methods

Sentence Transformers + FAISS for efficient search; + hard negative mining for better learning.

---

## Summary & Key Takeaways

Sentence Transformers generate effective semantic embeddings.

Principles:
1. Siamese networks: Parallel processing.
2. Triplet loss: Ordering constraint.
3. Contrastive learning: Similarity focus.
4. Pooling: Sentence representation.
5. Scalability: Efficient retrieval.

---

## Appendix: Practical Labs

### Lab 1: Triplet Loss

import numpy as np

def triplet_loss(anchor, positive, negative, margin=1.0):
 """Compute triplet loss"""
 pos_dist = np.linalg.norm(anchor - positive)
 neg_dist = np.linalg.norm(anchor - negative)
 
 loss = max(pos_dist - neg_dist + margin, 0)
 return loss

np.random.seed(42)
anchor = np.random.randn(768)
positive = anchor + np.random.randn(768) * 0.1
negative = np.random.randn(768)

loss = triplet_loss(anchor, positive, negative)
assert loss >= 0
print(f"✓ Triplet loss: {loss:.3f}")

### Lab 2: Contrastive Loss

import numpy as np

def contrastive_loss(similarities, labels, temperature=0.07):
 """Compute contrastive loss (InfoNCE)"""
 # Normalize similarities
 logits = similarities / temperature
 
 # Log-softmax
 log_probs = logits - np.log(np.sum(np.exp(logits), axis=-1, keepdims=True) + 1e-8)
 
 # Loss
 loss = -np.mean(log_probs[np.arange(len(labels)), labels])
 return loss

np.random.seed(42)
sims = np.random.randn(4, 8)
labels = np.array([1, 3, 2, 0])
loss = contrastive_loss(sims, labels)
assert loss > 0
print(f"✓ Contrastive loss: {loss:.3f}")

### Lab 3: Mean Pooling

import numpy as np

def mean_pooling(token_embeddings, attention_mask=None):
 """Extract sentence representation via mean pooling"""
 if attention_mask is not None:
 # Mask out padding tokens
 token_embeddings = token_embeddings * attention_mask[:, :, np.newaxis]
 sum_embeddings = np.sum(token_embeddings, axis=1)
 sum_mask = np.sum(attention_mask, axis=1, keepdims=True)
 else:
 sum_embeddings = np.sum(token_embeddings, axis=1)
 sum_mask = token_embeddings.shape[1]
 
 sentence_emb = sum_embeddings / sum_mask
 return sentence_emb

np.random.seed(42)
tokens = np.random.randn(4, 10, 768) # batch, seq_len, dim
mask = np.ones((4, 10))
mask[:, 5:] = 0 # Padding

sent_emb = mean_pooling(tokens, mask)
assert sent_emb.shape == (4, 768)
print("✓ Mean pooling working")

### Lab 4: Similarity Matrix

import numpy as np

def similarity_matrix(embeddings, metric='cosine'):
 """Compute similarity matrix"""
 if metric == 'cosine':
 embeddings = embeddings / (np.linalg.norm(embeddings, axis=1, keepdims=True) + 1e-8)
 similarities = embeddings @ embeddings.T
 elif metric == 'euclidean':
 dists = np.cdist(embeddings, embeddings, metric='euclidean')
 similarities = 1 / (1 + dists)
 
 return similarities

np.random.seed(42)
embs = np.random.randn(5, 768)
sims = similarity_matrix(embs)
assert sims.shape == (5, 5)
assert np.allclose(np.diag(sims), 1.0, atol=0.01)
print("✓ Similarity matrix working")

---