Custom Strategy Scripts

Custom strategy scripts give you full control over how your prompts are modified for adversarial testing by writing your own JavaScript files. This allows you to create completely custom red team testing approaches by transforming pre-existing test cases programmatically. Scripts can range from simple text transformations to calling external APIs or models.

:::info This page covers custom strategy scripts. For the built-in custom strategy that uses text-based instructions, see Custom Strategy. :::

Implementation

Use it in your promptfooconfig.yaml like this:

strategies:
  - id: file://custom-strategy.js
    config:
      optionalConfigKey: 'optionalConfigValue'

How It Works

Custom strategy scripts work by:

1. Defining a JavaScript module with an action function
2. Processing an array of test cases with your custom logic
3. Returning transformed test cases with new content
4. Tracking the transformation with metadata

Example Strategy Script

Here's a simple strategy script that ignores previous instructions:

module.exports = {
  id: 'ignore-previous-instructions',

  action: async (testCases, injectVar, config) => {
    return testCases.map((testCase) => ({
      ...testCase,
      vars: {
        ...testCase.vars,
        [injectVar]: `Ignore previous instructions: ${testCase.vars[injectVar]}`,
      },
      metadata: {
        ...testCase.metadata,
        strategyId: 'ignore-previous-instructions',
      },
    }));
  },
};

:::note Note that the strategy adds strategyId to the metadata while preserving the original pluginId using the spread operator (...testCase.metadata). Both identifiers are important for tracking and analysis purposes. :::

Configuration Options

The strategy action function receives:

• testCases: Array of test cases to transform. By default, this is the entire test suite. You can filter in your strategy implementation to specific plugins, etc.
• injectVar: Variable name to modify in each test case
• config: Optional configuration passed from promptfooconfig.yaml

Related Concepts

• Custom Strategy - Built-in customizable strategy using text-based instructions
• Strategy Development - Build custom approaches using JavaScript for maximum flexibility
• Test Case Transformation - Programmatically modify test cases to create unique attack vectors
• Red Team Strategies - Full strategy catalog