retrieval augmented generation production

**RAG in Production: Evaluation Frameworks and System Design — grounding LLM outputs in external knowledge to reduce hallucinations** Retrieval-Augmented Generation (RAG) augments LLM prompts with retrieved context (documents, web search), reducing hallucinations and enabling knowledge base updates without retraining. Production deployment requires evaluation frameworks, architecture patterns, and observability. **RAGAS Evaluation Framework** RAGAS (Retrieval-Augmented Generation Assessment): automated evaluation metrics sidestepping expensive human annotation. Metrics: (1) Faithfulness: does answer follow retrieved context? Decompose answer into claims, verify against context. (2) Answer Relevance: does answer address query? Reformulate answer as query, compute similarity. (3) Context Precision: is retrieved context relevant? Ratio of relevant documents to total retrieved. (4) Context Recall: did retrieval find all relevant documents? (requires ground truth labels, less practical). Scores: 0-1 scale, aggregatable over datasets. **Hallucination Characterization** LLM hallucinations: factually incorrect, contradicting context, or inventing references. RAG reduces (doesn't eliminate) hallucinations by providing grounding. Measurement: human evaluation vs. LLM-as-judge (cheaper but imperfect). Trustworthy hallucination detection: factual verification against retrieved documents (self-consistency scoring). **Hybrid Retrieval Pipelines** BM25 (bag-of-words, TF-IDF): sparse, interpretable, fast, no embeddings. Dense retrieval (embedding similarity): neural encoders capture semantic relationships, but slower than BM25. Hybrid: query→BM25 (fast k-1) + dense embedding (search k-2)→fuse rankings (reciprocal rank fusion). Re-ranking: initial retrieval→cross-encoder re-ranker→top-k. Cross-encoder (BERT-based) scores query-document pairs directly (more expensive but more accurate). **Chunk Size and Context Management** Document chunking: fixed size (512 tokens) vs. semantic (split on section boundaries). Trade-off: small chunks (precise matching, high retrieval count, token budget filled quickly) vs. large chunks (context coherence, fewer chunks to retrieve). Optimal: 256-1024 tokens, semantic boundaries. Context window management: limited prompt budget (8K-200K tokens)—retrieve top-k chunks fitting within budget. **RAG vs. Fine-Tuning Decision Framework** RAG: dynamic knowledge (add documents, update immediately), interpretability (retrieved docs show evidence), scaling (no retraining). Cost: retrieval latency (~100-500 ms), context contamination (noisy retrieval). Fine-tuning: static knowledge (requires retraining), lower latency (no retrieval), but expensive retraining, knowledge blending harder to verify. Decision: use RAG for frequently-updated knowledge (news, corporate documents); fine-tuning for stable, frequently-needed knowledge (domain-specific patterns, in-distribution data). **Enterprise Deployment** Architecture: document ingestion→chunking→embedding→vector DB (Weaviate, Pinecone, Milvus)→query→retrieval→LLM prompt→response. Security: document access control (index documents per user), query auditing, data retention. Observability: logs retrieval, LLM latency, costs. Caching: embed queries, cache popular results (5-10% of queries repeat); memoization of LLM outputs (same query within window→cached response).