LLM Evaluation Framework 2026: How to Test Prompts Like OpenAI Does (Promptfoo Guide)

Your prompt works once, then fails on the next input. You need a repeatable way to test LLM outputs before they reach users.

## LLM Evaluation Framework 2026: Stop Shipping Untested Prompts The fastest way to tell if an LLM change is good is not to eyeball it - it is to run a test suite. Promptfoo is the open-source LLM evaluation framework that does exactly this, and as of August 2026 it has 23,818 GitHub stars with active releases every week. OpenAI and Anthropic engineers are listed among its users, which tells you it survived real production use. ## What a Prompt Evaluation Actually Looks Like A promptfoo config is a YAML file with three parts: providers, prompts, and tests. You point it at an LLM API (OpenAI, Anthropic, Gemini, DeepSeek, or a local vLLM endpoint), list your prompt variants, then write assertions like contains: refund policy or similar: the API key belongs in the header. Run npx promptfoo eval and you get a side-by-side table of pass/fail rates per prompt. Change one system prompt and the number moves. That is the whole value: you now measure prompt quality instead of guessing. ## The Three Checks That Catch 80% of Regressions 1. **Exact-match assertions** - the output must contain a critical phrase (prices, dates, policy names). 2. **Similarity thresholds** - embedding-based comparison against a reference answer, useful for summarization. 3. **LLM-as-judge** - a stronger model scores helpfulness. Promptfoo ships with prebuilt judges so you do not need to write scoring prompts from scratch. ## Red Teaming Built In Since 2025 promptfoo has folded in AI red-teaming: it can generate adversarial inputs (jailbreak attempts, prompt injections) and check whether your guardrails hold. This turned an evaluation tool into a security tool, which is why security teams now run it in CI on every prompt change. ## FAQ **Is promptfoo free?** Yes, the core eval engine is MIT-licensed open source. A hosted cloud dashboard is optional. **Can it test local models?** Yes, it supports Ollama and any OpenAI-compatible endpoint. **How long does setup take?** A first config with 3 prompts and 10 test cases takes about 15 minutes. **Does it replace human review?** No. It catches regressions; humans still decide what quality means.