search
**Information Retrieval and Search** is the **field of systems that find and rank relevant information from large collections in response to user queries** — forming the backbone of search engines, enterprise knowledge bases, and retrieval-augmented generation (RAG) pipelines that ground AI systems in factual, up-to-date information.
**What Is Search and Retrieval?**
- **Definition**: Systems that, given a user query, efficiently find and rank the most relevant documents, passages, or records from a corpus potentially containing millions or billions of items.
- **Query Types**: Keyword queries ("TSMC 3nm process"), natural language questions ("What is the yield of N3 process?"), semantic queries (meaning-based), or structured queries (SQL, filters).
- **Evaluation Metrics**: Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG), Recall@K, Precision@K — measuring how well the relevant document ranks among retrieved results.
- **Scale**: Google indexes 100B+ web pages; enterprise search handles millions of internal documents; RAG systems retrieve from thousands to millions of passages.
**Why Search and Retrieval Matters**
- **Knowledge Access**: Enable users to find relevant information in seconds across vast document collections — from web search to scientific literature to enterprise wikis.
- **RAG Foundation**: Retrieval-augmented generation uses search to supply LLMs with relevant context — enabling AI systems to answer questions about current events and proprietary data without hallucination.
- **E-Commerce**: Product search and recommendation systems directly drive revenue — 1% improvement in search relevance can yield millions in revenue for large platforms.
- **Legal & Compliance**: Retrieve relevant case law, contracts, and regulatory documents for legal research and compliance verification.
- **Customer Support**: Find relevant help articles, past tickets, and product documentation to resolve customer issues quickly.
**Keyword Search — The Classical Foundation**
**TF-IDF (Term Frequency–Inverse Document Frequency)**:
- Score = (how often term appears in document) × (how rare the term is across all documents).
- Rare terms in a matching document signal high relevance; common words ("the", "is") get near-zero weight.
- Fast, interpretable, no training required — but literal matching only; "car" and "automobile" are unrelated.
**BM25 (Best Match 25)**:
- Probabilistic improvement over TF-IDF with term frequency saturation (diminishing returns for repeated terms) and document length normalization.
- Industry standard for keyword search — used in Elasticsearch, Lucene, and all major search engines as baseline.
- Parameters: k1 (term frequency saturation, typically 1.2–2.0), b (length normalization, typically 0.75).
**Inverted Index**:
- Data structure mapping each term to the list of documents containing it — enables O(log n) term lookup across billion-document corpora.
- Foundation of all keyword search systems.
**Semantic Search — Neural Retrieval**
**Bi-Encoder (Dense Retrieval)**:
- Encode query and documents separately into dense vectors using BERT-based encoders.
- Retrieve by approximate nearest-neighbor search (FAISS, HNSW, ScaNN) in vector space.
- Captures semantic similarity — "vehicle" and "car" are near neighbors in embedding space.
- Training: contrastive learning on (query, relevant document, negative document) triplets.
- Models: DPR (Dense Passage Retrieval), E5, BGE, Cohere Embed, OpenAI text-embedding-3.
**Cross-Encoder (Reranking)**:
- Jointly encode query + document through a single model — captures fine-grained interactions.
- Much more accurate than bi-encoders; 10–100x slower — used only for reranking top-K candidates.
**ColBERT (Late Interaction)**:
- Compute token-level embeddings for query and document independently, then score via MaxSim (maximum similarity per query token).
- Balance between bi-encoder speed and cross-encoder accuracy.
**RAG Search Pipeline**
**Step 1 — Indexing**: Chunk documents into passages (128–512 tokens), embed with bi-encoder, store in vector database (Pinecone, Weaviate, pgvector, Chroma).
**Step 2 — Retrieval**: Given query, embed with same encoder, retrieve top-K passages via ANN search (typically K=20–100).
**Step 3 — Reranking**: Cross-encoder reranks top-K to top-5 — improving precision at the cost of latency.
**Step 4 — Generation**: LLM generates response conditioned on retrieved context + original query.
**Retrieval System Comparison**
| Method | Accuracy | Speed | Semantic? | Infrastructure |
|--------|----------|-------|-----------|----------------|
| BM25 | Moderate | Very fast | No | Elasticsearch |
| Bi-encoder | Good | Fast (ANN) | Yes | Vector DB |
| Hybrid (BM25+dense) | Better | Fast | Partial | Both |
| Cross-encoder | Best | Slow | Yes | GPU inference |
| ColBERT | Good | Moderate | Yes | ColBERT index |
Search and retrieval is **the information access layer that determines whether AI systems answer from knowledge or hallucinate** — as hybrid retrieval systems combining keyword precision with semantic understanding become standard, high-quality grounded AI applications will scale to every enterprise knowledge domain.