embedding model

**Open Source Embedding Models (E5, BGE)** challenge proprietary models like OpenAI's by offering state-of-the-art performance on retrieval benchmarks (MTEB) while being free to run locally. **Key Models** **1. BGE (BAAI General Embedding)** - **Performance**: Consistently tops the MTEB leaderboard. - **Variants**: available in large, base, and small sizes. - **Instruction-tuned**: Requires specific prefix instructions for queries vs. passages. **2. E5 (Microsoft)** - **Method**: Text Embeddings by Weakly-Supervised Contrastive Pre-training. - **Quality**: Strong performance on zero-shot retrieval tasks. - **Format**: uses "query:" and "passage:" prefixes. **Comparison** - **OpenAI Ada-002**: Context length 8192, Pay-per-token, closed source. - **BGE-Large-en**: Context length 512 (v1.5 supports longer), Free, Open Weights, Local privacy. **Use Cases** - **Local RAG**: Privacy-preserving document search without external APIs. - **Cost Reduction**: Replacing paid embedding APIs for high-volume indexing. - **Custom Fine-tuning**: Can be fine-tuned on domain-specific data (unlike closed APIs).