Eval Guides

Practical tutorials to help you get the most out of promptfoo evals. Each guide walks through a real-world scenario with working configuration examples you can adapt to your own use case.

Evaluation Techniques

Learn how to measure specific aspects of LLM output quality.

• Evaluating RAG Pipelines — Test retrieval-augmented generation end-to-end
• Preventing Hallucinations — Measure and reduce hallucination rates with perplexity, RAG, and controlled decoding
• Evaluating JSON Outputs — Validate structured LLM outputs against schemas
• Evaluating Coding Agents — Test AI coding assistants
• Evaluating OSWorld with Inspect — Run desktop computer-use benchmark tasks through Inspect
• Testing Agent Skills — Compare Claude and Codex skill versions with routing, quality, cost, latency, and trace checks
• Evaluating Factuality — Score responses for factual accuracy
• LLM as a Judge — Build reliable model-graded evals with rubrics, calibration, and multi-judge checks
• Testing LLM Chains — Evaluate multi-step LLM pipelines
• Choosing the Right Temperature — Find the optimal temperature setting for your use case
• Evaluating Text-to-SQL — Measure SQL generation accuracy
• Sandboxed Code Evaluations — Safely run and test LLM-generated code
• HLE Benchmark — Evaluate models on the Humanity's Last Exam benchmark

Safety and Guardrails

Test provider safety controls and application-level enforcement.

• Testing and Validating Guardrails — Normalize guardrail responses, test attacks and false positives, and compare implementations
• Testing Google Cloud Model Armor — Evaluate Model Armor prompt and response templates
• Automated Red Teaming — Generate adversarial cases and measure application vulnerabilities

Model Comparisons

Compare LLM performance on your own data to make informed model selection decisions.

• GPT vs Claude vs Gemini — Benchmark the leading commercial models side-by-side
• DeepSeek Benchmark — Evaluate DeepSeek against leading models
• Choosing the Best GPT Model — Pick the right GPT variant for your workload
• GPT-5.2 vs o3 — Compare standard vs reasoning models
• Comparing Open-Source Models — Evaluate Mistral, Llama, Gemma, and Phi on custom datasets
• GPT Model Tiers MMLU-Pro — Measure GPT model tier performance on MMLU-Pro
• Qwen vs Llama vs GPT — Run a custom benchmark across model families
• OpenAI vs Azure Benchmark — Test whether Azure-hosted models match OpenAI directly
• Mixtral vs GPT — Pit Mixtral against GPT on real tasks
• Censored vs Uncensored Models — Compare content filtering behavior with Ollama

Integrations

Use promptfoo with popular frameworks and services.

• Using LangChain PromptTemplate — Integrate LangChain prompt templates with promptfoo
• Evaluating OpenAI Assistants — Evaluate OpenAI's Assistants API
• Evaluating CrewAI Agents — Red team and evaluate CrewAI multi-agent workflows
• Evaluating LangGraph — Test LangGraph agent applications
• Evaluating Google ADK Agents — Trace sessions, tools, callbacks, plugins, and workflow agents
• Evaluating ElevenLabs Voice AI — Test text-to-speech and voice AI outputs