RAG in Production: Evaluation Frameworks and System Design — grounding LLM outputs in external knowledge to reduce hallucinations

Retrieval-Augmented Generation (RAG) augments LLM prompts with retrieved context (documents, web search), reducing hallucinations and enabling knowledge base updates without retraining. Production deployment requires evaluation frameworks, architecture patterns, and observability.

RAGAS Evaluation Framework

RAGAS (Retrieval-Augmented Generation Assessment): automated evaluation metrics sidestepping expensive human annotation. Metrics: (1) Faithfulness: does answer follow retrieved context? Decompose answer into claims, verify against context. (2) Answer Relevance: does answer address query? Reformulate answer as query, compute similarity. (3) Context Precision: is retrieved context relevant? Ratio of relevant documents to total retrieved. (4) Context Recall: did retrieval find all relevant documents? (requires ground truth labels, less practical). Scores: 0-1 scale, aggregatable over datasets.

Hallucination Characterization

LLM hallucinations: factually incorrect, contradicting context, or inventing references. RAG reduces (doesn't eliminate) hallucinations by providing grounding. Measurement: human evaluation vs. LLM-as-judge (cheaper but imperfect). Trustworthy hallucination detection: factual verification against retrieved documents (self-consistency scoring).

Hybrid Retrieval Pipelines

BM25 (bag-of-words, TF-IDF): sparse, interpretable, fast, no embeddings. Dense retrieval (embedding similarity): neural encoders capture semantic relationships, but slower than BM25. Hybrid: query→BM25 (fast k-1) + dense embedding (search k-2)→fuse rankings (reciprocal rank fusion). Re-ranking: initial retrieval→cross-encoder re-ranker→top-k. Cross-encoder (BERT-based) scores query-document pairs directly (more expensive but more accurate).

Chunk Size and Context Management

Document chunking: fixed size (512 tokens) vs. semantic (split on section boundaries). Trade-off: small chunks (precise matching, high retrieval count, token budget filled quickly) vs. large chunks (context coherence, fewer chunks to retrieve). Optimal: 256-1024 tokens, semantic boundaries. Context window management: limited prompt budget (8K-200K tokens)—retrieve top-k chunks fitting within budget.

RAG vs. Fine-Tuning Decision Framework

RAG: dynamic knowledge (add documents, update immediately), interpretability (retrieved docs show evidence), scaling (no retraining). Cost: retrieval latency (~100-500 ms), context contamination (noisy retrieval). Fine-tuning: static knowledge (requires retraining), lower latency (no retrieval), but expensive retraining, knowledge blending harder to verify. Decision: use RAG for frequently-updated knowledge (news, corporate documents); fine-tuning for stable, frequently-needed knowledge (domain-specific patterns, in-distribution data).

Enterprise Deployment

Architecture: document ingestion→chunking→embedding→vector DB (Weaviate, Pinecone, Milvus)→query→retrieval→LLM prompt→response. Security: document access control (index documents per user), query auditing, data retention. Observability: logs retrieval, LLM latency, costs. Caching: embed queries, cache popular results (5-10% of queries repeat); memoization of LLM outputs (same query within window→cached response).