retrieval augmented generation rag

**Retrieval-Augmented Generation (RAG)** is the **AI architecture that enhances large language model responses by retrieving relevant information from external knowledge sources at inference time — grounding LLM outputs in factual, up-to-date, and domain-specific documents rather than relying solely on parametric knowledge baked in during training, dramatically reducing hallucinations and enabling enterprise deployment without costly fine-tuning**. **Why RAG Exists** LLMs have a knowledge cutoff date (training data stops at a point in time) and cannot access proprietary or real-time information. Fine-tuning is expensive, slow, and creates a new static snapshot. RAG solves both problems by retrieving relevant context dynamically at query time. **RAG Pipeline Architecture** **Indexing Phase (Offline)**: - **Document Ingestion**: Load documents from various sources (PDFs, databases, APIs, wikis). - **Chunking**: Split documents into semantically coherent chunks (256-1024 tokens). Strategies: fixed-size with overlap, recursive character splitting, semantic chunking (split at topic boundaries using embeddings). - **Embedding**: Encode each chunk into a dense vector using an embedding model (OpenAI text-embedding-3, BGE, GTE, E5). - **Vector Store**: Index embeddings in a vector database (Pinecone, Weaviate, Qdrant, FAISS, Chroma) with metadata (source, date, section). **Query Phase (Online)**: - **Query Embedding**: Encode the user query into the same embedding space. - **Retrieval**: Approximate nearest neighbor search returns top-K relevant chunks (typically K=3-10). - **Context Assembly**: Retrieved chunks are formatted into a prompt with the user query. - **Generation**: LLM generates a response grounded in the retrieved context. **Advanced RAG Techniques** - **Hybrid Search**: Combine dense vector search with sparse BM25 keyword search using Reciprocal Rank Fusion. Captures both semantic similarity and exact keyword matches. - **Query Transformation**: Rewrite the user query for better retrieval — HyDE (Hypothetical Document Embeddings) generates a hypothetical answer and uses it as the search query. Multi-query generates multiple reformulations and merges results. - **Re-Ranking**: After initial retrieval, a cross-encoder re-ranks the top-K chunks by relevance. Cohere Rerank, BGE-reranker, and ColBERT provide significant precision improvement. - **Agentic RAG**: The LLM decides when and what to retrieve through tool-calling — routing queries to different knowledge bases, performing multi-step retrieval, and synthesizing across sources. **Chunking Strategy Impact** Chunk size directly affects retrieval quality: too small (128 tokens) loses context continuity; too large (2048 tokens) dilutes relevance with irrelevant surrounding text. Optimal chunk size depends on document structure and query types — technical documentation benefits from larger chunks preserving procedure steps; FAQ-style content benefits from smaller, self-contained chunks. **Evaluation Metrics** - **Retrieval Quality**: Precision@K, Recall@K, NDCG — does the retriever find the right chunks? - **Generation Quality**: Faithfulness (is the answer supported by retrieved context?), relevance (does the answer address the query?), completeness. - **RAGAS Framework**: Automated evaluation using LLM-as-judge for faithfulness, answer relevance, and context relevance. RAG is **the pragmatic bridge between LLM capabilities and real-world knowledge requirements** — enabling organizations to deploy AI assistants that answer questions accurately from their own documents, without the cost and data requirements of fine-tuning, while maintaining the conversational fluency of foundation models.