embedding
**Embeddings** are **dense vector representations that capture semantic meaning of text, images, or other data** — transforming words, sentences, or documents into fixed-dimensional numerical vectors where similar concepts are closer in the vector space, enabling semantic search, clustering, classification, and retrieval-augmented generation (RAG).
**What Are Embeddings?**
- **Definition**: Learned numerical representations of data in vector space.
- **Property**: Similar items have similar vectors (close by distance).
- **Dimensions**: Typically 384-3072 floating point values.
- **Use**: Foundation for semantic search, RAG, similarity comparison.
**Why Embeddings Matter**
- **Semantic Understanding**: Find similar meaning, not just matching keywords.
- **Cross-Modal**: Compare text to text, images, or even audio.
- **Compression**: Dense representation of meaning.
- **Foundation**: Enable vector databases, RAG, recommendations.
- **Transfer Learning**: Pre-computed representations reusable across tasks.
**Embedding Levels**
**Word Embeddings** (Legacy):
- Word2Vec, GloVe: One vector per word.
- Same vector regardless of context.
- "Bank" has same embedding in "river bank" and "savings bank."
**Contextual Embeddings** (Modern):
- BERT, transformer-based: Different vectors based on context.
- "Bank" differs in "river bank" vs. "savings bank."
- Captures nuance and polysemy.
**Sentence/Document Embeddings**:
- Entire text chunk → single vector.
- Sentence-BERT, E5, BGE models.
- Used for semantic search, RAG.
**Popular Embedding Models**
```
Model | Dimensions | Use Case | Provider
-------------------|------------|--------------------|-----------
text-embedding-3 | 256-3072 | General purpose | OpenAI
E5-v2 | 1024 | Retrieval | Microsoft
BGE-v2 | 1024 | Multilingual | BAAI
Cohere Embed v3 | 1024 | Enterprise | Cohere
all-MiniLM-L6 | 384 | Fast, lightweight | SBERT
GTE | 768-1024 | General purpose | Alibaba
```
**How Embeddings Work**
```svg
```
**Similarity Metrics**
**Cosine Similarity** (most common):
```
cos(A, B) = (A · B) / (|A| × |B|)
Range: -1 to 1 (typically 0 to 1 for text)
Higher = more similar
```
**Euclidean Distance (L2)**:
```
L2(A, B) = sqrt(Σ(ai - bi)²)
Lower = more similar
Works best with normalized vectors
```
**Dot Product**:
```
dot(A, B) = Σ(ai × bi)
Higher = more similar
Equivalent to cosine for normalized vectors
```
**Embedding Applications**
**Semantic Search**:
```
Query: "machine learning tutorials for beginners"
↓ embed
Query Vector: [...]
↓ similarity search
Similar docs: [doc_47, doc_123, doc_89, ...]
```
**RAG (Retrieval-Augmented Generation)**:
```
1. User question → embed
2. Find similar knowledge chunks
3. Inject into LLM context
4. Generate grounded response
```
**Clustering/Classification**:
```
1. Embed all documents
2. Run clustering (K-means, HDBSCAN)
3. Discover topic groups automatically
```
**Duplicate Detection**:
```
1. Embed all items
2. Find pairs with similarity > threshold
3. Mark as likely duplicates
```
**Embedding Best Practices**
- **Match Model to Use Case**: Retrieval models for search, general for clustering.
- **Consistent Processing**: Same tokenization, truncation at query and index time.
- **Batch Processing**: GPU embedding is much faster in batches.
- **Dimension Trade-off**: Higher dims = more expressive, more memory/compute.
- **Quantization**: Store as int8 or binary for memory efficiency.
Embeddings are **the bridge between human language and machine computation** — by converting meaning into numbers, embeddings enable all the semantic AI applications that find "similar" rather than "exact" matches, making them foundational to modern AI systems.