A bi-encoder is a neural retrieval architecture that uses separate encoder networks to independently encode the query and document into dense vector representations. Similarity is then computed by comparing these vectors, typically using cosine similarity or dot product.

How Bi-Encoders Work

• Document Encoding (Offline): All documents in the corpus are pre-encoded into vectors and stored in an index (typically a vector database). This is a one-time cost.
• Query Encoding (Online): At search time, the query is encoded into a vector using the query encoder.
• Retrieval: The query vector is compared against all document vectors using approximate nearest neighbor (ANN) search, returning the most similar documents in milliseconds.

Advantages

• Speed: Since documents are pre-encoded, retrieval only requires encoding the query and performing a fast vector lookup — sub-millisecond latency for millions of documents.
• Scalability: Works efficiently with corpora of billions of documents using ANN indexes like HNSW or IVF.
• Independence: Query and document encoders can be based on different model architectures if needed.

Bi-Encoder vs. Cross-Encoder

• Bi-Encoder: Fast but less accurate — query and document never "see" each other during encoding, so fine-grained token-level interactions are missed.
• Cross-Encoder: Processes query+document together through a single model, capturing rich interactions, but is 100–1000× slower since every candidate must be scored individually.
• Common Pattern: Use a bi-encoder for first-stage retrieval (fast, broad recall) followed by a cross-encoder for reranking the top results (slow, high precision).

Popular Bi-Encoder Models

• Sentence-BERT (SBERT)
• E5 and BGE families
• GTE (General Text Embeddings)
• Cohere Embed and OpenAI text-embedding-3

Bi-encoders are the backbone of modern semantic search and RAG retrieval systems.