Sentence Transformers are neural network models that produce fixed-length embeddings for sentences and paragraphs — enabling semantic similarity search, clustering, and retrieval by mapping semantically related texts to nearby points in embedding space.

The Core Problem

• BERT produces contextualized token embeddings — not a single sentence representation.
• Naive [CLS] token: Poor for semantic similarity (requires fine-tuning).
• Naive mean pooling: Better but still suboptimal.
• SBERT: Fine-tune with siamese/triplet networks → excellent sentence embeddings.

Sentence-BERT (SBERT) Architecture

• Siamese BERT: Two identical BERT models processing sentence pairs.
• Mean-pooled output → fixed-size sentence vector.
• Trained with: Natural Language Inference (NLI) data + triplet/cosine objectives.
• Cosine similarity of SBERT embeddings correlates strongly with human semantic judgment.

Training Objectives

• Cosine Similarity Loss: Minimize angle between positive pairs; maximize for negative pairs.
• Multiple Negative Ranking (MNR): In-batch negatives — scale efficiently.
• Triplet Loss: $|sim(a,p) - sim(a,n)| > \epsilon$ — anchor closer to positive than negative.

Bi-Encoder vs. Cross-Encoder

RAG Retrieval Stack

• Bi-encoder: Retrieve top-100 from vector DB (milliseconds).
• Cross-encoder: Rerank top-100 → top-5 (100ms).
• Combine both for optimal quality/speed tradeoff.

Key Models

• all-MiniLM-L6-v2: 22M params, 384-dim, very fast — popular for production.
• BGE-large (Beijing Academy): Best MTEB score in open-source (mid-2024).
• E5-mistral-7b: LLM-based embeddings — top accuracy but expensive.
• OpenAI text-embedding-3-large: 3072-dim, top accuracy for SaaS.

Sentence transformers are the foundation of modern semantic search and RAG systems — their ability to compress arbitrary text into searchable vectors at millisecond speed is what makes LLM-powered knowledge bases and retrieval systems practical at scale.