embedding model retrieval

**Text Embedding Models for Retrieval** are **neural networks that map text passages of arbitrary length to fixed-dimensional dense vectors where semantic similarity is captured by vector proximity (cosine similarity or dot product) — enabling sub-second semantic search over millions of documents by replacing keyword matching with meaning-based matching, powering RAG systems, recommendation engines, and semantic search applications**. **Why Dense Retrieval Outperforms Keyword Search** Traditional search (BM25, TF-IDF) matches exact terms — a query for "how to fix a flat tire" won't match a document about "repairing a punctured wheel." Dense retrieval encodes both query and document into vectors where semantically equivalent texts have high cosine similarity regardless of word choice, capturing synonymy, paraphrase, and conceptual similarity. **Architecture** - **Bi-Encoder**: Separate encoders for query and document (or shared encoder). Each text is independently encoded to a vector. Similarity = dot_product(q_vec, d_vec). Documents can be pre-encoded and indexed. At query time, only the query needs encoding. Standard for production systems. - **Cross-Encoder**: Both query and document are concatenated and processed jointly through a single model. More accurate (full cross-attention between query and document tokens) but requires processing every query-document pair at search time — too slow for first-stage retrieval but excellent as a reranker. **Training** - **Contrastive Learning**: The embedding model is trained to maximize similarity between (query, positive_document) pairs and minimize similarity with negative documents. The InfoNCE loss pulls positive pairs together and pushes hard negatives apart. - **Hard Negative Mining**: Random negatives are too easy. Effective training requires hard negatives — documents that are superficially similar to the query but not actually relevant. Mined from BM25 results or from the embedding model's own retrieval. - **Knowledge Distillation**: Cross-encoder scores are distilled into bi-encoder training, using the cross-encoder's superior relevance judgments as soft labels. **Indexing and Search** - **HNSW (Hierarchal Navigable Small World)**: The dominant approximate nearest neighbor (ANN) index. Builds a hierarchical proximity graph enabling ~90% recall at <1ms latency for 1M+ vectors. Libraries: FAISS, Milvus, Qdrant, Pinecone. - **IVF (Inverted File Index)**: Clusters vectors into Voronoi cells. At query time, searches only the nearest clusters. Trading recall for speed. - **Quantization (PQ, SQ)**: Compress vectors from 768×float32 (3KB) to 96 bytes via Product Quantization, enabling billion-scale indexes in memory. **Key Models** - **E5 / BGE / GTE**: Open-source embedding models trained on massive retrieval datasets. 768-1024 dimensional vectors. State-of-the-art on MTEB benchmarks. - **OpenAI text-embedding-3-large**: Commercial embedding model with adjustable dimensionality (256-3072). Text Embedding Models are **the neural compression that maps the infinite space of human language into geometric points where meaning defines distance** — enabling machines to find relevant information not by matching words but by understanding intent.