braintrust
**Braintrust** is an **enterprise-grade AI evaluation platform that integrates LLM quality testing directly into the development and CI/CD workflow** — providing a dataset management system, prompt playground, and automated regression testing framework that treats "did this prompt change break my use case?" as a first-class engineering question with a quantitative answer.
**What Is Braintrust?**
- **Definition**: A commercial AI evaluation and observability platform (founded 2023) that combines logging, dataset management, prompt experimentation, and automated evaluation into a unified workflow — enabling engineering teams to apply the same rigor to LLM quality as they apply to software testing.
- **CI/CD Integration**: Braintrust evaluations run as code — Python or TypeScript eval scripts that execute in CI pipelines, compare results against a baseline score, and fail the build if quality regresses beyond a threshold.
- **Dataset Versioning**: Test cases are stored as versioned datasets — curated from production logs, hand-labeled examples, or synthetic data — and every evaluation run is linked to the exact dataset version used.
- **Scoring System**: Define custom scoring functions (exact match, semantic similarity, LLM-as-judge, human review) that evaluate any aspect of your application's output quality.
- **Prompt Playground**: Iterate on prompts against your dataset in a browser UI, see scores update in real-time, and promote the best version to production with full audit trail.
**Why Braintrust Matters**
- **Catching Regressions Before Production**: When a developer changes a system prompt to fix one issue, Braintrust runs the full evaluation suite and alerts if other use cases degrade — preventing the "fix one thing, break another" cycle that plagues LLM application development.
- **Evidence-Based Decisions**: Model upgrades (e.g., GPT-4o-mini → GPT-4o) are evaluated quantitatively across your actual use cases before committing — cost/quality tradeoffs become data-driven decisions.
- **Production Data Loop**: Real user interactions are automatically logged and can be curated into test cases — the evaluation dataset grows organically from production usage, continuously covering new edge cases.
- **Multi-Metric Evaluation**: A single LLM response can be scored simultaneously on accuracy, groundedness, safety, tone, and latency — giving a multi-dimensional view of quality changes.
- **Enterprise Readiness**: SOC 2 compliant, SSO support, team permissions, and audit logs — meets enterprise security requirements for regulated industries.
**Core Braintrust Workflow**
**Defining an Evaluation**:
```python
import braintrust
from braintrust import Eval
async def my_task(input):
response = openai.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": input["question"]}]
)
return response.choices[0].message.content
async def accuracy_scorer(output, expected):
return 1.0 if output.strip().lower() == expected.strip().lower() else 0.0
Eval(
"Customer Support QA",
data=[{"input": {"question": "What is your return policy?"}, "expected": "30-day returns"}],
task=my_task,
scores=[accuracy_scorer]
)
```
**Running in CI**:
```bash
braintrust eval my_eval.py --threshold 0.85
# Fails CI if average score drops below 85%
```
**Key Braintrust Features**
**Logging**:
- Wrap any LLM call with `braintrust.traced()` to capture inputs, outputs, latency, tokens, and cost.
- Every production request is logged and searchable — find the exact trace behind a user complaint.
**Experiments**:
- Compare two prompt versions side-by-side with statistical significance testing.
- "Version B is 12% more accurate than Version A with p < 0.05" — confidence before deployment.
**Datasets**:
- Build test suites from production logs, manual curation, or synthetic generation.
- Version datasets separately from code — reproduce any historical evaluation exactly.
**Human Review**:
- Route uncertain cases to human reviewers in the Braintrust UI.
- Collect human labels that improve automated scorer calibration over time.
**Braintrust vs Alternatives**
| Feature | Braintrust | Langfuse | Promptfoo | LangSmith |
|---------|-----------|---------|----------|----------|
| CI/CD integration | Excellent | Good | Excellent | Good |
| Dataset management | Strong | Strong | Good | Strong |
| Enterprise focus | Very high | Medium | Low | Medium |
| Open source | No | Yes | Yes | No |
| Human review workflow | Strong | Good | Limited | Good |
| Multi-metric scoring | Strong | Good | Good | Strong |
Braintrust is **the evaluation platform that makes LLM quality regression testing as reliable and automated as unit testing in traditional software development** — for engineering teams that need quantitative answers to "did this change make my AI worse?", Braintrust provides the infrastructure to catch quality regressions before they reach users.