Promptfoo is an open-source command-line tool for systematically testing and evaluating LLM prompts across multiple models and providers — enabling developers to define test cases in YAML, run them against OpenAI, Anthropic, Ollama, and any other provider simultaneously, and get quantitative scores that replace "vibes-based" prompt engineering with data-driven iteration.
What Is Promptfoo?
- Definition: An open-source CLI tool and library (MIT license, 4,000+ GitHub stars) that runs structured evaluations of LLM prompts — taking test case inputs, running them through one or more models, applying scoring assertions (regex match, LLM-as-judge, semantic similarity, custom Python/JavaScript functions), and producing a comparison report.
- YAML-First Configuration: Evaluations are defined in a promptfooconfig.yaml file — prompts, providers, test cases, and assertions are all declarative, making evaluations version-controllable and reproducible.
- Multi-Provider Testing: Run the same prompt through GPT-4o, Claude 3.5 Sonnet, Llama-3, and a local Ollama model in a single command — compare quality and cost across providers simultaneously.
- Assertion Types: Built-in assertions include exact string match, regex, cosine similarity, LLM-based quality scoring (LLM-as-judge), and arbitrary JavaScript/Python evaluation functions.
- CI/CD Integration: Runs as a CLI command (npx promptfoo eval) — integrates into GitHub Actions, GitLab CI, or any pipeline to catch prompt regressions automatically.
Why Promptfoo Matters
- Systematic vs Ad-Hoc Testing: Most prompt development involves manually trying a few examples and deciding "that looks good." Promptfoo forces definition of test cases upfront and evaluates them all consistently — the same discipline software testing brings to code.
- Multi-Model Comparison: Evaluating GPT-4o vs Claude 3.5 Haiku on your specific use case is one command — real performance data on your actual task replaces benchmark comparisons that may not generalize.
- Red Teaming: Built-in adversarial test generation for safety testing — promptfoo can automatically generate jailbreak attempts, prompt injection attacks, and bias-revealing inputs to identify vulnerabilities before deployment.
- Cost Visibility: Each test run reports token usage and estimated cost per provider — model selection becomes a cost/quality optimization with real numbers.
- Open Source and Self-Hosted: No data leaves your environment — test proprietary prompts without concerns about model providers training on your evaluation data.
Core Usage
Basic Configuration (promptfooconfig.yaml):
``yaml
prompts:
- "Summarize the following in one sentence: {{input}}"
- "Provide a concise one-sentence summary of: {{input}}"
providers:
- openai:gpt-4o
- anthropic:claude-3-5-haiku-20241022
- ollama:llama3
tests:
- vars:
input: "The quick brown fox jumps over the lazy dog near the riverbank."
assert:
- type: contains
value: "fox"
- type: llm-rubric
value: "Is the summary accurate and under 20 words?"
- vars:
input: "Quarterly earnings exceeded analyst expectations by 15% on strong cloud revenue."
assert:
- type: regex
value: "earnings|revenue|quarter"
`
Run with: npx promptfoo eval
Assertion Types
- contains: Response must include a specific substring — simple factual checks.
- regex: Response must match a regular expression — structured data extraction validation.
- llm-rubric: An LLM grades the response against a natural language criterion — flexible quality assessment.
- similar: Cosine similarity above threshold vs a reference answer — semantic correctness without exact match.
- javascript: Custom JavaScript function — any logic expressible in JS.
- python: Custom Python function — leverage any Python library for evaluation.
Red Teaming:
`yaml`
redteam:
plugins:
- harmful:hate # Test for hate speech generation
- jailbreak # Test prompt injection resistance
- pii:direct # Test PII leakage
strategies:
- jailbreak
- prompt-injection
CI/CD Integration:
`yaml``
# .github/workflows/eval.yml
- name: Run LLM Evals
run: npx promptfoo eval --ci
# Fails if any assertion fails — blocks PR merge
Promptfoo vs Alternatives
| Feature | Promptfoo | Braintrust | DeepEval | Langfuse |
|---------|----------|-----------|---------|---------|
| Open source | Yes (MIT) | No | Yes | Yes |
| CLI-first | Yes | No | Yes (pytest) | No |
| Multi-provider | Excellent | Good | Good | Good |
| Red teaming | Built-in | No | Limited | No |
| CI/CD integration | Excellent | Good | Good | Good |
| Setup time | Minutes | Hours | Hours | Hours |
Promptfoo is the open-source evaluation tool that brings test-driven development discipline to prompt engineering — by making it trivial to define test cases, run them across multiple models, and integrate evaluation into CI/CD, promptfoo enables any developer to replace subjective prompt quality judgments with objective, reproducible, data-driven iteration.