DeepEval is an open-source LLM evaluation framework that runs as pytest-compatible unit tests in CI/CD pipelines — providing pre-built metrics for hallucination detection, contextual relevance, bias, answer correctness, and G-Eval scoring that treat LLM quality as a testable, measurable property rather than a subjective judgment.

What Is DeepEval?

• Definition: An open-source Python evaluation framework (Confident AI, 2023) that integrates with pytest to define LLM quality tests — each test specifies an input, actual output, optional expected output, and retrieval context, then applies one or more metric objects that score the output and fail the test if the score falls below a threshold.
• Pytest Integration: Write assert_test(test_case, metrics) calls inside standard pytest functions — run deepeval test run and get a pytest-compatible test report, enabling LLM quality testing in any existing CI/CD system.
• Pre-Built Metrics: 14+ production-ready metrics covering the main dimensions of LLM quality — no custom metric code needed for common evaluation scenarios.
• LLM-as-Judge: Most DeepEval metrics use GPT-4 or another LLM to evaluate outputs — natural language criteria are more flexible than regex or exact match for complex quality dimensions.
• Confident AI Platform: Results automatically upload to Confident AI's dashboard for trend tracking, regression alerts, and team visibility — optional cloud layer on top of the open-source framework.

Why DeepEval Matters

• Shift Left Quality: Catching hallucinations or bias in a CI/CD pipeline before deployment is orders of magnitude cheaper than discovering them in production — DeepEval makes this possible with standard pytest tooling.
• Metric Standardization: Teams no longer need to define "what is a hallucination?" for their specific use case — DeepEval's Faithfulness metric provides a standardized, calibrated definition backed by research.
• RAG-Specific Coverage: The full RAG evaluation stack (retrieval quality, context precision, context recall, faithfulness, answer relevance) is covered by dedicated metrics — no need to piece together a custom evaluation framework.
• Regression Prevention: Pin expected minimum scores in test assertions — when a model update or prompt change causes hallucination rate to increase from 3% to 12%, the test fails and blocks deployment automatically.
• Research-Backed: Metrics are grounded in published LLM evaluation research (RAGAS, G-Eval, TruLens) with calibrated score interpretations.

Core DeepEval Metrics

Faithfulness (Hallucination Detection):

• Measures whether claims in the actual output are supported by the retrieval context.
• Score of 1.0 = fully grounded, 0.0 = entirely hallucinated.
• Uses an LLM to extract claims and verify each against provided context.

Contextual Precision (Retrieval Quality):

• Measures whether retrieved context nodes are relevant to the query.
• High precision = retrieved chunks are useful. Low = retriever is pulling irrelevant content.

Contextual Recall:

• Measures whether the retrieval context contains all information needed to answer the query.
• Low recall = retriever missed important documents — knowledge gap in the corpus.

Answer Relevancy:

• Measures whether the actual output addresses the input question.
• Catches responses that are factually correct but don't answer the question asked.

G-Eval (Flexible LLM Scoring):

• User-defined evaluation criteria specified in natural language.
• Example: "Score from 0-10 whether the response is professional and avoids jargon."

Bias and Toxicity:

• Detect discriminatory language, stereotyping, or toxic content in outputs.
• Critical for customer-facing applications serving diverse user populations.

Usage Example

import pytest
from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_rag_faithfulness():
    test_case = LLMTestCase(
        input="What is the return policy?",
        actual_output="Returns are accepted within 30 days with receipt.",
        retrieval_context=["Our policy: customers may return items within 30 days of purchase with proof of purchase."]
    )
    faithfulness = FaithfulnessMetric(threshold=0.8, model="gpt-4o")
    answer_relevancy = AnswerRelevancyMetric(threshold=0.7, model="gpt-4o")
    assert_test(test_case, [faithfulness, answer_relevancy])

Run with: deepeval test run test_rag.py

Bulk Evaluation:

from deepeval import evaluate

test_cases = [LLMTestCase(...) for _ in dataset]
results = evaluate(test_cases, metrics=[FaithfulnessMetric(threshold=0.8)])

DeepEval vs Alternatives

DeepEval is the evaluation framework that brings unit testing discipline to LLM application quality assurance — by making hallucination, relevance, and bias metrics runnable as pytest assertions in CI/CD pipelines, DeepEval enables engineering teams to catch quality regressions automatically and ship LLM applications with measurable, verifiable quality guarantees.