Embeddings are dense vector representations that capture semantic meaning of text, images, or other data — transforming words, sentences, or documents into fixed-dimensional numerical vectors where similar concepts are closer in the vector space, enabling semantic search, clustering, classification, and retrieval-augmented generation (RAG).

What Are Embeddings?

• Definition: Learned numerical representations of data in vector space.
• Property: Similar items have similar vectors (close by distance).
• Dimensions: Typically 384-3072 floating point values.
• Use: Foundation for semantic search, RAG, similarity comparison.

Why Embeddings Matter

• Semantic Understanding: Find similar meaning, not just matching keywords.
• Cross-Modal: Compare text to text, images, or even audio.
• Compression: Dense representation of meaning.
• Foundation: Enable vector databases, RAG, recommendations.
• Transfer Learning: Pre-computed representations reusable across tasks.

Embedding Levels

Word Embeddings (Legacy):

• Word2Vec, GloVe: One vector per word.
• Same vector regardless of context.
• "Bank" has same embedding in "river bank" and "savings bank."

Contextual Embeddings (Modern):

• BERT, transformer-based: Different vectors based on context.
• "Bank" differs in "river bank" vs. "savings bank."
• Captures nuance and polysemy.

Sentence/Document Embeddings:

• Entire text chunk → single vector.
• Sentence-BERT, E5, BGE models.
• Used for semantic search, RAG.

Popular Embedding Models

Model              | Dimensions | Use Case           | Provider
-------------------|------------|--------------------|-----------
text-embedding-3   | 256-3072   | General purpose    | OpenAI
E5-v2              | 1024       | Retrieval          | Microsoft
BGE-v2             | 1024       | Multilingual       | BAAI
Cohere Embed v3    | 1024       | Enterprise         | Cohere
all-MiniLM-L6      | 384        | Fast, lightweight  | SBERT
GTE                | 768-1024   | General purpose    | Alibaba

How Embeddings Work

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 470" width="100%" height="100%" style="background:#0F172A;font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,Helvetica,Arial,sans-serif;">
  <defs>
    <linearGradient id="cardGrad" x1="0%" y1="0%" x2="0%" y2="100%">
      <stop offset="0%" stop-color="#1E293B"/>
      <stop offset="100%" stop-color="#0F172A"/>
    </linearGradient>
  </defs>

  <!-- Title Banner -->
  <rect x="20" y="20" width="720" height="56" rx="14" fill="url(#cardGrad)" stroke="#334155" stroke-width="1"/>
  <text x="36" y="46" font-size="18" font-weight="800" fill="#F8FAFC">Vector Embeddings &amp; Semantic Search Space</text>
  <text x="36" y="64" font-size="12" font-weight="600" fill="#94A3B8">High-Dimensional Feature Mapping · Cosine Similarity Metric · Dense Vector Representation</text>

  <!-- Panel 1: Text to Dense Vector Transformation -->
  <rect x="20" y="90" width="350" height="350" rx="16" fill="url(#cardGrad)" stroke="#334155" stroke-width="1"/>
  <text x="40" y="118" font-size="14" font-weight="700" fill="#38BDF8">1. Dense Embedding Generation</text>

  <!-- Raw Input Text Box -->
  <rect x="40" y="140" width="310" height="48" rx="10" fill="#0F172A" stroke="#475569" stroke-width="1"/>
  <text x="55" y="160" font-size="11" font-weight="700" fill="#CBD5E1">Input Token Stream:</text>
  <text x="55" y="176" font-size="12" font-weight="800" fill="#38BDF8">"King"  |  "Queen"  |  "Apple"  |  "Fruit"</text>

  <!-- Encoder Arrow -->
  <path d="M 195,190 L 195,215" stroke="#94A3B8" stroke-width="2" marker-end="url(#arrow)"/>
  <text x="205" y="206" font-size="10" font-weight="700" fill="#94A3B8">Transformer Encoder / Projection</text>

  <!-- Vector Output Representation -->
  <rect x="40" y="220" width="310" height="95" rx="10" fill="#1E293B" stroke="#0284C7" stroke-width="1.5"/>
  <text x="55" y="242" font-size="12" font-weight="800" fill="#F8FAFC">Dense Vector Rᵈ (e.g. d=768 / 1536):</text>
  <text x="55" y="265" font-size="11" font-family="monospace" fill="#38BDF8">E("King")  = [+0.24, -0.81, +0.15, ...]</text>
  <text x="55" y="285" font-size="11" font-family="monospace" fill="#A78BFA">E("Queen") = [+0.22, -0.79, +0.18, ...]</text>
  <text x="55" y="305" font-size="11" font-family="monospace" fill="#F59E0B">E("Apple") = [-0.64, +0.12, +0.91, ...]</text>

  <!-- Key Concept Footnote -->
  <rect x="40" y="330" width="310" height="95" rx="10" fill="#0F172A" stroke="#334155" stroke-width="1"/>
  <text x="55" y="352" font-size="11" font-weight="700" fill="#F8FAFC">Semantic Property:</text>
  <text x="55" y="372" font-size="11" font-weight="600" fill="#CBD5E1">• Similar meanings mapping to close vector space coordinates.</text>
  <text x="55" y="392" font-size="11" font-weight="600" fill="#38BDF8">• Vector Arithmetic: E("King") - E("Man") + E("Woman") ≈ E("Queen")</text>

  <!-- Panel 2: Vector Space & Distance Metrics -->
  <rect x="390" y="90" width="350" height="350" rx="16" fill="url(#cardGrad)" stroke="#334155" stroke-width="1"/>
  <text x="410" y="118" font-size="14" font-weight="700" fill="#38BDF8">2. Vector Space &amp; Cosine Distance</text>

  <!-- Visual Coordinate System -->
  <rect x="410" y="135" width="310" height="175" rx="10" fill="#0F172A" stroke="#334155" stroke-width="1"/>
  <line x1="440" y1="280" x2="700" y2="280" stroke="#475569" stroke-width="1.5"/>
  <line x1="440" y1="280" x2="440" y2="150" stroke="#475569" stroke-width="1.5"/>
  
  <!-- Vector Points & Rays -->
  <!-- Vector A: King -->
  <line x1="440" y1="280" x2="620" y2="175" stroke="#38BDF8" stroke-width="2.5"/>
  <circle cx="620" cy="175" r="5" fill="#38BDF8"/>
  <text x="630" y="175" font-size="11" font-weight="800" fill="#38BDF8">King</text>

  <!-- Vector B: Queen -->
  <line x1="440" y1="280" x2="650" y2="195" stroke="#A78BFA" stroke-width="2.5"/>
  <circle cx="650" cy="195" r="5" fill="#A78BFA"/>
  <text x="660" y="195" font-size="11" font-weight="800" fill="#A78BFA">Queen</text>

  <!-- Vector C: Apple (Far away) -->
  <line x1="440" y1="280" x2="490" y2="170" stroke="#F59E0B" stroke-width="2.5"/>
  <circle cx="490" cy="170" r="5" fill="#F59E0B"/>
  <text x="495" y="165" font-size="11" font-weight="800" fill="#F59E0B">Apple</text>

  <!-- Angle arc between King & Queen -->
  <path d="M 480,257 A 40,40 0 0 1 488,262" fill="none" stroke="#F8FAFC" stroke-width="2"/>
  <text x="495" y="252" font-size="10" font-weight="700" fill="#F8FAFC">θ ≈ 12°</text>

  <!-- Metrics Box -->
  <rect x="410" y="325" width="310" height="100" rx="10" fill="#1E293B" stroke="#475569" stroke-width="1"/>
  <text x="425" y="348" font-size="12" font-weight="800" fill="#38BDF8">Similarity Formulas:</text>
  <text x="425" y="370" font-size="11" font-weight="600" fill="#F8FAFC">Cosine Sim:  cos(θ) = (A · B) / (||A|| ||B||)</text>
  <text x="425" y="392" font-size="11" font-weight="600" fill="#CBD5E1">Dot Product:  A · B  |  Euclidean:  ||A - B||₂</text>
  <text x="425" y="412" font-size="10" font-weight="700" fill="#F59E0B">Normalized vectors: Cosine Sim = Dot Product</text>
</svg>

Similarity Metrics

Cosine Similarity (most common):

cos(A, B) = (A · B) / (|A| × |B|)

Range: -1 to 1 (typically 0 to 1 for text)
Higher = more similar

Euclidean Distance (L2):

L2(A, B) = sqrt(Σ(ai - bi)²)

Lower = more similar
Works best with normalized vectors

Dot Product:

dot(A, B) = Σ(ai × bi)

Higher = more similar
Equivalent to cosine for normalized vectors

Embedding Applications

Semantic Search:

Query: "machine learning tutorials for beginners"
         ↓ embed
Query Vector: [...]
         ↓ similarity search
Similar docs: [doc_47, doc_123, doc_89, ...]

RAG (Retrieval-Augmented Generation):

1. User question → embed
2. Find similar knowledge chunks
3. Inject into LLM context
4. Generate grounded response

Clustering/Classification:

1. Embed all documents
2. Run clustering (K-means, HDBSCAN)
3. Discover topic groups automatically

Duplicate Detection:

1. Embed all items
2. Find pairs with similarity > threshold
3. Mark as likely duplicates

Embedding Best Practices

• Match Model to Use Case: Retrieval models for search, general for clustering.
• Consistent Processing: Same tokenization, truncation at query and index time.
• Batch Processing: GPU embedding is much faster in batches.
• Dimension Trade-off: Higher dims = more expressive, more memory/compute.
• Quantization: Store as int8 or binary for memory efficiency.

Embeddings are the bridge between human language and machine computation — by converting meaning into numbers, embeddings enable all the semantic AI applications that find "similar" rather than "exact" matches, making them foundational to modern AI systems.