Eval Guides

Practical tutorials to help you get the most out of promptfoo evals. Each guide walks through a real-world scenario with working configuration examples you can adapt to your own use case.

Evaluation Techniques

Learn how to measure specific aspects of LLM output quality.

• Evaluating RAG Pipelines — Test retrieval-augmented generation end-to-end
• Preventing Hallucinations — Measure and reduce hallucination rates with perplexity, RAG, and controlled decoding
• Evaluating JSON Outputs — Validate structured LLM outputs against schemas
• Evaluating Coding Agents — Test AI coding assistants
• Evaluating Factuality — Score responses for factual accuracy
• LLM as a Judge — Build reliable model-graded evals with rubrics, calibration, and multi-judge checks
• Testing LLM Chains — Evaluate multi-step LLM pipelines
• Choosing the Right Temperature — Find the optimal temperature setting for your use case
• Evaluating Text-to-SQL — Measure SQL generation accuracy
• Sandboxed Code Evaluations — Safely run and test LLM-generated code
• HLE Benchmark — Evaluate models on the Humanity's Last Exam benchmark

Model Comparisons

Compare LLM performance on your own data to make informed model selection decisions.

• GPT vs Claude vs Gemini — Benchmark the leading commercial models side-by-side
• DeepSeek Benchmark — Evaluate DeepSeek against leading models
• Choosing the Best GPT Model — Pick the right GPT variant for your workload
• GPT-5.2 vs o3 — Compare standard vs reasoning models
• Comparing Open-Source Models — Evaluate Mistral, Llama, Gemma, and Phi on custom datasets
• GPT Model Tiers MMLU-Pro — Measure GPT model tier performance on MMLU-Pro
• Qwen vs Llama vs GPT — Run a custom benchmark across model families
• OpenAI vs Azure Benchmark — Test whether Azure-hosted models match OpenAI directly
• Mixtral vs GPT — Pit Mixtral against GPT on real tasks
• Censored vs Uncensored Models — Compare content filtering behavior with Ollama

Integrations

Use promptfoo with popular frameworks and services.

• Using LangChain PromptTemplate — Integrate LangChain prompt templates with promptfoo
• Evaluating OpenAI Assistants — Evaluate OpenAI's Assistants API
• Evaluating CrewAI Agents — Red team and evaluate CrewAI multi-agent workflows
• Evaluating LangGraph — Test LangGraph agent applications
• Evaluating ElevenLabs Voice AI — Test text-to-speech and voice AI outputs