ragas
**RAGAS (RAG Assessment)** is an **open-source evaluation framework for measuring the quality of Retrieval Augmented Generation systems using reference-free LLM-as-judge metrics** — automatically scoring faithfulness, answer relevance, context precision, and context recall without requiring hand-labeled ground truth for every query, enabling continuous RAG quality monitoring at scale.
**What Is RAGAS?**
- **Definition**: An open-source Python library (Exploding Gradients, 2023) that evaluates RAG pipeline quality across four core dimensions — faithfulness, answer relevance, context precision, and context recall — using LLMs as evaluators rather than requiring human-labeled reference answers for every test case.
- **Reference-Free Evaluation**: The key innovation of RAGAS is evaluating without ground truth labels — it uses an evaluation LLM to judge whether the answer is supported by the retrieved context, whether the context is relevant, and whether the answer addresses the question — making it practical to evaluate thousands of production queries.
- **Four Core Metrics**: Together, the four RAGAS metrics form a comprehensive quality picture — diagnose whether failures come from the retriever (context quality) or the generator (answer quality).
- **Integration**: Works with LangChain, LlamaIndex, and any custom RAG pipeline — output is a pandas DataFrame with per-query scores suitable for aggregation, visualization, and CI/CD thresholding.
- **Dataset Generation**: RAGAS can automatically generate evaluation datasets (question-context-answer triples) from a document corpus using an LLM — eliminating the manual work of creating test cases.
**Why RAGAS Matters**
- **Holistic RAG Debugging**: A RAG system has two components — retriever and generator. When quality is poor, RAGAS tells you which component is responsible: low context precision/recall → fix the retriever; low faithfulness/answer relevance → fix the generator or prompt.
- **No Labels Required**: Creating ground truth labels for 10,000 production queries is impractical. RAGAS makes it possible to evaluate quality across your entire production log without any labeling cost.
- **Continuous Monitoring**: Run RAGAS nightly on a sample of production queries — track metric trends over time and alert when faithfulness drops (suggesting knowledge base staleness or model degradation).
- **A/B Evaluation**: Compare two RAG configurations (different chunk sizes, embedding models, or LLMs) on the same query set with RAGAS scores — objective evidence for architectural decisions.
- **Research Grounding**: RAGAS metrics are grounded in published research with calibration studies showing strong correlation with human quality judgments.
**The Four RAGAS Metrics Explained**
**Faithfulness** (Generator Quality — Hallucination Detection):
- *"Does the answer contain only claims that are supported by the retrieved context?"*
- Process: LLM extracts factual claims from the answer, then verifies each claim against the retrieved context.
- Score 1.0 = every claim is grounded in context. Score 0.0 = answer is entirely fabricated.
- Low faithfulness → generator is hallucinating beyond the provided context. Fix: stronger grounding prompt, smaller temperature.
**Answer Relevance** (Generator Quality — On-Topic):
- *"Does the answer actually address the question that was asked?"*
- Process: Evaluation LLM generates hypothetical questions that the answer would address, then measures cosine similarity to the original question.
- Low score = answer is factually correct but doesn't answer the specific question. Fix: prompt engineering or question reformulation.
**Context Precision** (Retriever Quality — Signal-to-Noise):
- *"Are the retrieved chunks actually useful for answering the question?"*
- Process: LLM evaluates whether each retrieved chunk contains relevant information, weighted by rank position.
- Low precision = retriever is returning irrelevant documents mixed with relevant ones. Fix: better embedding model, metadata filtering, or reranker.
**Context Recall** (Retriever Quality — Completeness):
- *"Does the retrieved context contain all the information needed to answer the question?"*
- Process: LLM checks whether each sentence in the ground truth answer can be attributed to the retrieved context.
- Low recall = retriever is missing relevant documents. Fix: retrieve more chunks (higher k), improve chunk splitting, or enrich knowledge base.
**Usage Example**
```python
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset
data = {
"question": ["What is the return policy?"],
"answer": ["Returns are accepted within 30 days."],
"contexts": [["Items can be returned within 30 days of purchase with a receipt."]],
"ground_truth": ["Returns are allowed within 30 days with proof of purchase."]
}
dataset = Dataset.from_dict(data)
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision, context_recall])
print(result)
# faithfulness: 0.97, answer_relevancy: 0.94, context_precision: 1.00, context_recall: 0.92
```
**Dataset Generation**:
```python
from ragas.testset.generator import TestsetGenerator
generator = TestsetGenerator.with_openai()
testset = generator.generate_with_langchain_docs(documents, test_size=100)
```
**RAGAS vs Alternatives**
| Feature | RAGAS | DeepEval | TruLens | Human Eval |
|---------|------|---------|--------|-----------|
| Reference-free | Yes | Yes | Yes | No |
| RAG-specific metrics | Excellent | Good | Good | N/A |
| Dataset generation | Yes | No | No | No |
| LangChain integration | Native | Good | Good | N/A |
| Research backing | Strong | Strong | Strong | Gold standard |
| Scale | Excellent | Good | Good | Poor |
RAGAS is **the evaluation framework that makes systematic RAG quality measurement practical at production scale** — by providing reference-free metrics that use LLMs as judges, RAGAS enables teams to continuously monitor their retrieval and generation quality across thousands of queries without the prohibitive cost of human labeling.