hybrid search
**Hybrid Search: Combining BM25 and Dense Retrieval**
**Retrieval Methods**
**Sparse Retrieval (BM25)**
Traditional keyword matching with term frequency weighting:
- Fast and efficient
- Works well for exact matches and keywords
- No semantic understanding
- Handles rare terms well
**Dense Retrieval (Vector Search)**
Semantic similarity using embeddings:
- Understands meaning and synonyms
- Better for natural language queries
- May miss exact keyword matches
- Requires embedding infrastructure
**Why Hybrid?**
Neither method is perfect alone. Hybrid search combines strengths:
| Query Type | BM25 | Dense | Hybrid |
|------------|------|-------|--------|
| Exact keyword | Strong | Weak | Strong |
| Semantic concept | Weak | Strong | Strong |
| Rare terms | Strong | Weak | Strong |
| Synonyms | Weak | Strong | Strong |
**Hybrid Implementation**
```python
def hybrid_search(query: str, alpha: float = 0.5) -> list:
# BM25 scores
bm25_results = bm25_index.search(query, top_k=100)
bm25_scores = normalize(bm25_results.scores)
# Dense scores
query_embedding = embed(query)
dense_results = vector_store.search(query_embedding, top_k=100)
dense_scores = normalize(dense_results.scores)
# Combine scores
combined = {}
for doc_id, score in bm25_results:
combined[doc_id] = alpha * score
for doc_id, score in dense_results:
combined[doc_id] = combined.get(doc_id, 0) + (1 - alpha) * score
# Sort by combined score
return sorted(combined.items(), key=lambda x: x[1], reverse=True)
```
**Reciprocal Rank Fusion (RRF)**
Another combination method using ranks instead of scores:
```python
def rrf_score(rank: int, k: int = 60) -> float:
return 1 / (k + rank)
def rrf_fusion(results_list: list) -> dict:
scores = {}
for results in results_list:
for rank, doc_id in enumerate(results):
scores[doc_id] = scores.get(doc_id, 0) + rrf_score(rank)
return scores
```
**Tools with Hybrid Search**
| Tool | Hybrid Support |
|------|----------------|
| Elasticsearch 8+ | Native |
| Weaviate | Native |
| Qdrant | Built-in sparse vectors |
| Pinecone | Via sparse-dense |
| Vespa | Native |
**Tuning Alpha**
| Query Pattern | Recommended Alpha |
|---------------|-------------------|
| Keyword-heavy | 0.7 (more BM25) |
| Conversational | 0.3 (more dense) |
| Balanced | 0.5 |
Tune alpha on your specific dataset and query patterns.