retrieval augmented generation
**Retrieval-Augmented Generation (RAG)** is the **architecture pattern that enhances LLM responses by first retrieving relevant documents from an external knowledge base and injecting them into the prompt context before generation** — grounding the model's output in factual, up-to-date information rather than relying solely on parametric knowledge learned during pretraining, dramatically reducing hallucination and enabling domain-specific expertise.
**RAG Pipeline**
1. **User Query**: "What is the maximum operating temperature for the XYZ-3000 chip?"
2. **Retrieval**: Query embedded → similarity search against vector database → top-K relevant documents retrieved.
3. **Augmentation**: Retrieved documents prepended to the prompt as context.
4. **Generation**: LLM generates answer grounded in the retrieved context.
**Components**
| Component | Purpose | Examples |
|-----------|---------|----------|
| Embedding Model | Convert text to vectors | OpenAI ada-002, BGE, E5, GTE |
| Vector Database | Store and search embeddings | Pinecone, Weaviate, Qdrant, Chroma, pgvector |
| Chunking Strategy | Split documents into retrievable units | 256-1024 tokens, overlap 10-20% |
| Reranker | Re-score retrieved results for relevance | Cohere Rerank, BGE-reranker, cross-encoder |
| LLM | Generate final answer from context | GPT-4, Claude, LLaMA, Mistral |
**Chunking Strategies**
| Strategy | Chunk Size | Overlap | Best For |
|----------|-----------|---------|----------|
| Fixed-size | 512 tokens | 50 tokens | General text |
| Sentence-based | 3-5 sentences | 1 sentence | Precise retrieval |
| Semantic | Variable | None | Coherent topics |
| Recursive character | 1000 chars | 200 chars | LangChain default |
| Parent-child | Small retrieve, large return | N/A | Best of both worlds |
**Advanced RAG Techniques**
- **Hybrid Search**: Combine vector similarity with keyword (BM25) search → better recall.
- **HyDE**: Generate hypothetical answer first → use it as retrieval query → better embedding match.
- **Multi-Query**: LLM generates multiple query variations → retrieve for each → union results.
- **Reranking**: Initial retrieval (fast, approximate) → reranker scores (slow, accurate) → top results.
- **Agentic RAG**: LLM decides when and what to retrieve iteratively — not just single retrieval.
**Evaluation Metrics**
| Metric | What It Measures |
|--------|----------------|
| Faithfulness | Is the answer supported by retrieved context? |
| Answer Relevance | Does the answer address the question? |
| Context Relevance | Are retrieved documents relevant to the query? |
| Context Recall | Did retrieval find all necessary information? |
**RAG vs. Fine-Tuning**
- **RAG**: Dynamic knowledge, no retraining needed, traceable sources, handles knowledge updates.
- **Fine-tuning**: Bakes knowledge into weights, better for style/format changes, no retrieval latency.
- **Best practice**: Use both — fine-tune for domain style, RAG for factual knowledge.
RAG is **the standard architecture for production LLM applications** — by separating knowledge storage (database) from reasoning (LLM), it solves the core limitations of LLMs: outdated knowledge, hallucination, and lack of domain expertise, making it essential for enterprise AI deployments.