Retrieval-Augmented Generation (RAG) is the architecture pattern that enhances LLM responses by first retrieving relevant documents from an external knowledge base and injecting them into the prompt context — grounding the model's generation in factual, up-to-date, and source-attributable information rather than relying solely on parametric knowledge memorized during training.

Why RAG Is Necessary

LLMs hallucinate because they generate text based on statistical patterns, not verified facts. Their training data has a knowledge cutoff date, and they cannot access proprietary or real-time information. RAG solves all three problems: retrieved documents provide factual grounding, the knowledge base can be continuously updated, and answers can cite specific sources.

The RAG Pipeline

1. Indexing (Offline): Documents are split into chunks (typically 256-1024 tokens), each chunk is converted to a dense vector embedding using an embedding model (e.g., text-embedding-3-large, BGE, E5), and the embeddings are stored in a vector database (Pinecone, Weaviate, Qdrant, pgvector). 2. Retrieval (Online): The user query is embedded with the same model. A similarity search (cosine similarity or approximate nearest neighbor) finds the top-K most relevant chunks from the vector store. 3. Augmentation: Retrieved chunks are prepended to the user query in the LLM prompt, typically with instructions like "Answer the question based on the following context." 4. Generation: The LLM generates a response grounded in the retrieved context, ideally citing which chunks support each claim.

Chunking Strategies

• Fixed-Size: Split by token count with overlap windows (e.g., 512 tokens, 50-token overlap). Simple but may break semantic boundaries.
• Semantic Chunking: Split at natural boundaries (paragraphs, sections, sentences) to preserve meaning within each chunk.
• Recursive/Hierarchical: Create both fine-grained (paragraph) and coarse-grained (section/document) chunks. Retrieve at the fine level, expand to the coarse level for context.

Advanced RAG Techniques

• Hybrid Search: Combine dense vector retrieval with sparse keyword retrieval (BM25) using reciprocal rank fusion for more robust recall.
• Re-Ranking: A cross-encoder reranker (e.g., Cohere Rerank, BGE-reranker) scores each retrieved chunk against the query with full cross-attention, improving precision over embedding-only similarity.
• Query Transformation: Rewrite the user query (expansion, decomposition, HyDE — hypothetical document embeddings) to improve retrieval quality.
• Agentic RAG: The LLM decides when and what to retrieve, iteratively refining queries based on initial results, and reasoning over multi-hop information chains.

Evaluation Metrics

• Faithfulness: Does the generated answer contradict the retrieved context?
• Answer Relevancy: Does the answer address the user question?
• Context Precision/Recall: Did retrieval find the right chunks?

Retrieval-Augmented Generation is the practical bridge between LLM fluency and factual accuracy — turning language models from impressive but unreliable text generators into grounded, source-backed knowledge systems.