site/docs/releases.md
Full release history for Promptfoo open source can be found on GitHub.
This month we shipped adaptive rate limiting, Transformers.js for local inference, telecom red team plugins, and video generation providers.
apiBaseUrl, websocketUrl override, and function call support__count variable - Use in derived metrics for computing averagesmetadata.sessionId surfaced as a variable column in tables and exportspromptfoo logs - New command for viewing log files directly$VAR syntax in file paths--env-file flags - Load multiple environment files--share and --no-share to control cloud sharingRed team scans now support multiple input variables, allowing you to test systems with complex input structures.
-d/--description flag - Add descriptions to redteam generate commands-j flag now propagates to Python worker poolsThis month we shipped video generation providers, OWASP Agentic AI Top 10, xAI Voice Agent, and multi-modal attack strategies.
--extension CLI flag - Load extensions via command lineevaluate() APIThis month we shipped Hydra multi-turn strategy, code scanning, Claude Opus 4.5, and VS Code extension.
Hydra is a new advanced multi-turn red team strategy that adapts dynamically based on target responses, using conversation techniques to probe for vulnerabilities.
Code scanning analyzes your codebase for potential AI security issues before they reach production.
Install the VS Code red team extension to run security scans directly from your editor.
This month we shipped jailbreak:meta red team strategy, remediation reports, and Postman/cURL import for HTTP targets.
eq, neq, gt, gte, lt, lte, and is_defined operatorsRemediation reports include:
Access from any vulnerability report by clicking "View Remediation Report".
jailbreak:meta uses multiple AI agents to generate attacks. This single-shot strategy is up to 50% more effective than some multi-turn attacks.
Plugin-specific grading rules:
This month we shipped reusable custom policies, risk scoring, 8 new AI providers, and comprehensive enterprise features for security teams.
web_fetch_20250910 and web_search_20250305 tool supportazure:responses provider aliasUse Ctrl+C to pause long-running evaluations and promptfoo eval --resume to continue later.
promptfoo export logs creates tar.gz for debuggingCustom policies can now be saved to a library and reused across red team evaluations:
Reference policies in your red team config:
redteam:
plugins:
- id: policy
config:
policy: 'internal-customer-data-protection'
Red team reports now include quantitative risk scores based on severity, probability, and impact:
Use risk scores to prioritize remediation and set CI/CD deployment gates.
<|im_start|>, <|im_end|>)This month we added support for new models, model audit cloud sharing, and performance improvements.
Model audit results can now be shared to the cloud for team collaboration:
Added support for advanced authentication methods:
Added auto-fill capabilities for HTTP provider setup to reduce configuration time and errors.
Added Medical Off-Label Use Detection plugin to identify inappropriate pharmaceutical recommendations that could endanger patients.
Added Unverifiable Claims Detection plugin to test AI systems' susceptibility to generating fabricated but plausible-sounding claims.
Added MCP Agent example for red team testing with tool call results, demonstrating how to test AI systems that use Model Context Protocol.
This month we focused on expanding provider support, enhancing evaluation capabilities, and strengthening enterprise features to help you build more reliable and secure AI applications.
import pdb; pdb.set_trace() in executed third-party Python scripts for easier debuggingWe've significantly improved the evaluation results interface to handle large-scale testing more effectively:
First-Class Zooming Support - Zoom in and out of the eval results table to see more data at once or focus on specific details. This is especially useful when working with evaluations containing hundreds or thousands of test cases.
Advanced Metadata Filtering - Filter results using powerful search operators (equals, contains, not contains) with persistent button actions. Click on any metric pill in the results to instantly apply it as a filter, making it easier to drill down into specific failure modes or success patterns.
Improved Pagination - Enhanced pagination controls with "go to" functionality and better handling of large result sets. The UI now maintains scroll position and filter state as you navigate between pages.
Multi-Metric Filtering - Apply multiple filters simultaneously to find exactly the results you're looking for. For red team evaluations, you can now filter by both plugin and strategy to analyze specific attack vectors.
Performance Optimizations - Fixed horizontal scrolling issues, improved rendering performance for large tables, and optimized memory usage when dealing with extensive evaluation results.
These improvements make it much easier to analyze and understand evaluation results, especially for large-scale red teaming exercises or comprehensive test suites.
We've launched two powerful new agentic multi-turn red team strategies that adapt dynamically based on target responses:
Custom Strategy - Define your own red teaming strategies using natural language instructions. This groundbreaking feature lets you create sophisticated, domain-specific attack patterns without writing code. The AI agent interprets your instructions and executes multi-turn conversations tailored to your specific testing needs.
Mischievous User Strategy - Simulates an innocently mischievous user who plays subtle games with your AI agent through multi-turn conversations. This strategy uncovers vulnerabilities by mimicking real-world user behavior where users might push boundaries through playful or indirect approaches rather than direct attacks.
Both strategies leverage AI agents to conduct intelligent, adaptive conversations that evolve based on your system's responses, making them far more effective than static attack patterns.
This month we focused on enhancing observability, expanding provider support, and strengthening red team capabilities to help you build more reliable and secure AI applications.
We've added OpenTelemetry tracing support to help you understand what's happening inside your AI applications. Previously, LLM applications were often "black boxes"—you could see inputs and outputs, but not what happened in between. Now you can visualize the entire execution flow, measure performance of individual steps, and quickly identify issues.
This is especially valuable for complex RAG pipelines or multi-step workflows where you need to identify performance bottlenecks or debug failures.
Use it when:
As AI applications increasingly use voice interfaces and visual content, you need tools to evaluate these capabilities just as rigorously as text-based interactions. We've significantly expanded support for audio and multimodal AI:
Google Live Audio - Full audio generation with features like:
Hyperbolic Provider - New support for Hyperbolic's image and audio models, providing more options for multimodal evaluations
Helicone AI Gateway - Route requests through Helicone for enhanced monitoring and analytics
Mistral Magistral - Added support for Mistral's latest reasoning models
Supply chain attacks through compromised models are a growing threat. We've significantly enhanced our static model security scanner to help you verify model integrity before deployment, checking for everything from malicious pickle files to subtle statistical anomalies that might indicate trojaned models.
New Web Interface: ModelAudit now includes a visual UI accessible at /model-audit when running promptfoo view:
Expanded Format Support:
.bin files (PyTorch, SafeTensors, etc.)Security Improvements:
PROMPTFOO_MAX_EVAL_TIME_MS environment variable prevents runaway evaluations from consuming excessive resourcesGeneric attacks often miss system-specific vulnerabilities. We've added powerful features for organizations that need sophisticated AI security testing to create targeted tests that match your actual security risks:
Target Discovery Agent - Automatically analyzes your AI system to understand its capabilities and craft more effective, targeted attacks
Adaptive Red Team Strategies - Define complex multi-turn attack strategies with enhanced capabilities for targeted testing
Grader Customization - Fine-tune evaluation criteria at the plugin level with concrete examples for more accurate assessments
Cloud-based Plugin Severity Overrides - Enterprise users can centrally manage and customize severity levels for red team plugins across their organization
Different industries face unique AI risks. We've introduced specialized plugins for industries where AI errors have serious consequences, ensuring you're testing for the failures that matter most in your domain:
Medical Plugins detect critical healthcare risks:
Financial Plugins identify domain-specific vulnerabilities:
Biased AI systems can perpetuate discrimination at scale. Our new comprehensive bias detection tests ensure your AI treats all users fairly and respectfully across:
The Intent (Custom Prompts) plugin now supports JSON file uploads with nested arrays for multi-step attack sequences. The enhanced UI makes it easier to manage complex test scenarios.
Red team tests now include automatic token estimation for HTTP providers, helping you track costs even with custom API integrations.
A new System Prompt Override plugin tests whether your LLM deployment is vulnerable to system instruction manipulation—a critical security flaw that could disable safety features.
Real attacks rarely succeed in a single message. We've enhanced our attack strategies to better simulate how bad actors actually try to manipulate AI systems through extended, adaptive conversations:
Enhanced GOAT and Crescendo - Now include intelligent agents that can:
Emoji Encoding Strategy - New obfuscation technique using emoji to bypass content filters