retrieval augmented generation rag

**Retrieval-Augmented Generation (RAG)** is the **architecture pattern that enhances LLM responses by first retrieving relevant documents from an external knowledge base and injecting them into the prompt context — grounding the model's generation in factual, up-to-date, and source-attributable information rather than relying solely on parametric knowledge memorized during training**. **Why RAG Is Necessary** LLMs hallucinate because they generate text based on statistical patterns, not verified facts. Their training data has a knowledge cutoff date, and they cannot access proprietary or real-time information. RAG solves all three problems: retrieved documents provide factual grounding, the knowledge base can be continuously updated, and answers can cite specific sources. **The RAG Pipeline** 1. **Indexing (Offline)**: Documents are split into chunks (typically 256-1024 tokens), each chunk is converted to a dense vector embedding using an embedding model (e.g., text-embedding-3-large, BGE, E5), and the embeddings are stored in a vector database (Pinecone, Weaviate, Qdrant, pgvector). 2. **Retrieval (Online)**: The user query is embedded with the same model. A similarity search (cosine similarity or approximate nearest neighbor) finds the top-K most relevant chunks from the vector store. 3. **Augmentation**: Retrieved chunks are prepended to the user query in the LLM prompt, typically with instructions like "Answer the question based on the following context." 4. **Generation**: The LLM generates a response grounded in the retrieved context, ideally citing which chunks support each claim. **Chunking Strategies** - **Fixed-Size**: Split by token count with overlap windows (e.g., 512 tokens, 50-token overlap). Simple but may break semantic boundaries. - **Semantic Chunking**: Split at natural boundaries (paragraphs, sections, sentences) to preserve meaning within each chunk. - **Recursive/Hierarchical**: Create both fine-grained (paragraph) and coarse-grained (section/document) chunks. Retrieve at the fine level, expand to the coarse level for context. **Advanced RAG Techniques** - **Hybrid Search**: Combine dense vector retrieval with sparse keyword retrieval (BM25) using reciprocal rank fusion for more robust recall. - **Re-Ranking**: A cross-encoder reranker (e.g., Cohere Rerank, BGE-reranker) scores each retrieved chunk against the query with full cross-attention, improving precision over embedding-only similarity. - **Query Transformation**: Rewrite the user query (expansion, decomposition, HyDE — hypothetical document embeddings) to improve retrieval quality. - **Agentic RAG**: The LLM decides when and what to retrieve, iteratively refining queries based on initial results, and reasoning over multi-hop information chains. **Evaluation Metrics** - **Faithfulness**: Does the generated answer contradict the retrieved context? - **Answer Relevancy**: Does the answer address the user question? - **Context Precision/Recall**: Did retrieval find the right chunks? Retrieval-Augmented Generation is **the practical bridge between LLM fluency and factual accuracy** — turning language models from impressive but unreliable text generators into grounded, source-backed knowledge systems.