Reference · Glossary
promptfoo
Last updated
A CLI/config-driven tool for **testing and comparing prompts** — you write a YAML file of test cases and assertions, run it against one or more models, and get a pass/fail grid plus a diff view, including built-in red-team checks for jailbreaks and prompt injection.
#When to use
Regression-testing a prompt before shipping a change, or comparing how the same prompt performs across two different models/providers side by side.
#When not to
Deep custom scoring logic that needs full Python (like RAGAS-style faithfulness scoring) — promptfoo's assertions cover common cases but a code-based framework gives more control for bespoke metrics.
#Example
prompts: ["Summarize: {{text}}"]
providers: ["openai:gpt-4o-mini", "anthropic:claude-3-5-sonnet"]
tests:
- vars: { text: "Long policy document..." }
assert:
- type: llm-rubric
value: "Summary is under 50 words and mentions the refund window"