sentence transformer
**Sentence Transformers** are **neural network models that produce fixed-length embeddings for sentences and paragraphs** — enabling semantic similarity search, clustering, and retrieval by mapping semantically related texts to nearby points in embedding space.
**The Core Problem**
- BERT produces contextualized token embeddings — not a single sentence representation.
- Naive [CLS] token: Poor for semantic similarity (requires fine-tuning).
- Naive mean pooling: Better but still suboptimal.
- SBERT: Fine-tune with siamese/triplet networks → excellent sentence embeddings.
**Sentence-BERT (SBERT) Architecture**
- Siamese BERT: Two identical BERT models processing sentence pairs.
- Mean-pooled output → fixed-size sentence vector.
- Trained with: Natural Language Inference (NLI) data + triplet/cosine objectives.
- Cosine similarity of SBERT embeddings correlates strongly with human semantic judgment.
**Training Objectives**
- **Cosine Similarity Loss**: Minimize angle between positive pairs; maximize for negative pairs.
- **Multiple Negative Ranking (MNR)**: In-batch negatives — scale efficiently.
- **Triplet Loss**: $|sim(a,p) - sim(a,n)| > \epsilon$ — anchor closer to positive than negative.
**Bi-Encoder vs. Cross-Encoder**
| Feature | Bi-Encoder | Cross-Encoder |
|---------|-----------|---------------|
| Architecture | Two separate encoders | Joint encoding of pair |
| Inference | Pre-compute embeddings | Must process pair together |
| Speed | Fast (vector search) | Slow (no precomputation) |
| Accuracy | Good | Better |
| Use case | First-stage retrieval | Reranking |
**RAG Retrieval Stack**
- Bi-encoder: Retrieve top-100 from vector DB (milliseconds).
- Cross-encoder: Rerank top-100 → top-5 (100ms).
- Combine both for optimal quality/speed tradeoff.
**Key Models**
- **all-MiniLM-L6-v2**: 22M params, 384-dim, very fast — popular for production.
- **BGE-large (Beijing Academy)**: Best MTEB score in open-source (mid-2024).
- **E5-mistral-7b**: LLM-based embeddings — top accuracy but expensive.
- **OpenAI text-embedding-3-large**: 3072-dim, top accuracy for SaaS.
Sentence transformers are **the foundation of modern semantic search and RAG systems** — their ability to compress arbitrary text into searchable vectors at millisecond speed is what makes LLM-powered knowledge bases and retrieval systems practical at scale.