Dense Retrieval and Embedding Models are the neural information retrieval systems that encode queries and documents into dense vector representations in a shared semantic space — enabling semantic search where relevance is measured by vector similarity rather than keyword overlap, finding conceptually related documents even with no shared vocabulary, powering applications from question answering systems to RAG pipelines and enterprise search.

Sparse vs Dense Retrieval

DPR (Dense Passage Retrieval)

• Karpukhin et al. (2020): Dual-encoder architecture for open-domain QA.
• Question encoder: BERT → 768-d vector for query.
• Passage encoder: Separate BERT → 768-d vector for document passage.
• Training: Contrastive loss — maximize similarity of (question, positive passage) pairs, minimize similarity to negatives.
• Retrieval: FAISS index over 21M Wikipedia passages → retrieve top-k by dot product.
• Key result: DPR significantly outperforms BM25 for natural language questions.

In-Batch Negatives Training

def contrastive_loss(q_embeds, p_embeds, temperature=0.07):
    # q_embeds: [B, D] query embeddings
    # p_embeds: [B, D] positive passage embeddings
    # Other passages in batch serve as hard negatives
    
    scores = torch.matmul(q_embeds, p_embeds.T) / temperature  # [B, B]
    labels = torch.arange(B)  # diagonal is positive pair
    return F.cross_entropy(scores, labels)

Sentence Transformers (SBERT)

• Siamese BERT: Encode two sentences → mean-pool → compare with cosine similarity.
• Fine-tuned on NLI (entailment pairs as positives, contradiction as negatives).
• Enables efficient semantic textual similarity (STS) → used for clustering, semantic search.
• SBERT is 9,000× faster than cross-encoder for ranking 10,000 sentences.

Modern Embedding Models

MTEB (Massive Text Embedding Benchmark)

• 56 tasks across 7 categories: Retrieval, classification, clustering, STS, reranking, etc.
• 112 languages → comprehensive multilingual evaluation.
• Standard leaderboard for comparing embedding models.

ANN (Approximate Nearest Neighbor) Search

• Exact k-NN over millions of vectors is too slow → approximate search.
• FAISS: Facebook AI similarity search → IVF (inverted file) + PQ (product quantization) → 100M vectors in < 10ms.
• HNSW: Hierarchical navigable small world graph → fast and accurate for moderate scales.
• ScaNN (Google): Optimized for TPU; state-of-the-art recall-latency trade-off.

Retrieval in RAG Pipelines

• Chunk documents → embed each chunk → store in vector database (Pinecone, Weaviate, Chroma).
• At query time: Embed query → retrieve top-k chunks by similarity → inject into LLM context.
• Hybrid retrieval: Combine dense score + BM25 score → better than either alone.
• Reranking: Cross-encoder rescores top-k retrieved passages → better precision at top positions.

Dense retrieval and embedding models are the semantic backbone of modern AI-powered search and knowledge retrieval — by learning that "cardiac arrest" and "heart attack" are semantically equivalent without sharing a single word, dense retrievers close the vocabulary gap that made keyword search frustrating for decades, enabling the retrieval-augmented generation pipelines that allow LLMs to access specialized knowledge bases, corporate documents, and up-to-date information far beyond what can fit in a context window.