Red Team Providers

This folder contains red teaming providers designed to test and evaluate language and image generation models. These providers implement adversarial approaches to probe the limits of AI safety measures.

Iterative Provider

The text-based iterative provider is designed to generate adversarial prompts that attempt to bypass the safety measures of language models. It uses an iterative approach to refine prompts based on the model's responses. It:

• Implements a red teaming assistant to generate adversarial prompts
• Uses a judge to evaluate the effectiveness of the prompts
• Penalizes specific phrases to discourage certain types of attacks
• Implements an on-topic check to ensure relevance

flowchart TB
    Start([Start]) --> Init[Initialize]
    Init --> Loop{Iteration
Loop}

    Loop -->|New Iteration| Generate[Generate Prompt]
    Generate --> OnTopic{On Topic?}

    OnTopic -->|Yes| Target[Send to Target Model]
    OnTopic -->|No| Feedback

    Target --> Judge[Judge Response]
    Judge --> Penalize{Contains
Penalized
Phrase?}

    Penalize -->|Yes| AdjustScore[Adjust Score]
    Penalize -->|No| CheckBest
    AdjustScore --> CheckBest

    CheckBest{Best Score?}
    CheckBest -->|Yes| UpdateBest[Update Best Response]
    CheckBest -->|No| Feedback

    UpdateBest --> Feedback[Prepare Feedback]
    Feedback --> CheckEnd{End Conditions:
1. Score ≥ 10
2. Max Iterations
Reached}

    CheckEnd -->|No| Loop
    CheckEnd -->|Yes| End([End])

    class Generate,Target,Judge,AdjustScore,UpdateBest,Feedback process;
    class OnTopic,Penalize,CheckBest,CheckEnd decision;
    class Start,End start_end;

Iterative Tree Provider

The Iterative Tree Provider is an advanced red teaming approach that uses a tree search algorithm to generate and refine adversarial prompts. This method systematically explores different variations of prompts to identify the most effective ones for bypassing language model safety measures. It:

• Initializes with a root node representing the initial prompt.
• Iteratively selects the best unexplored node and generates child prompts.
• Evaluates each prompt based on its relevance and effectiveness.
• Updates the tree with new nodes and their scores.
• Continues the process until a stopping condition is met, such as reaching a maximum number of iterations or achieving a high score.

graph TD
    A[Start] --> B[Initialize Root Node]
    B --> C[Tree Search Loop]
    C --> D[Select Best Unexplored Node]
    D --> E[Generate Child Prompts]
    E --> F[Evaluate Prompts]
    F --> G[Update Tree]
    G --> H{Stop Condition Met?}
    H -->|No| C
    H -->|Yes| I[Return Best Result]
    I --> J[End]

    subgraph Evaluation Process
    F --> F1[Check On-Topic]
    F1 --> F2[Get Model Response]
    F2 --> F3[Check Non-Refusal]
    F3 --> F4[Judge Response]
    F4 --> F5[Calculate Score]
    end

Key Features

• Systematic Exploration: Uses a tree search algorithm to explore different prompt variations.
• Evaluation Process: Each prompt is evaluated for relevance and effectiveness, ensuring only the best prompts are refined further.
• Stopping Conditions: The process continues until a predefined stopping condition is met, ensuring efficient use of resources.

IterativeImage Provider

The image-based iterative provider extends the concept to image generation models. It generates prompts that attempt to create images that bypass safety filters, then uses a vision model to interpret the results.

flowchart TB
    Start([Start]) --> Init[Initialize Providers]
    Init --> Loop{Iteration
Loop}

    Loop -->|New Iteration| Generate[Generate Prompt]
    Generate --> OnTopic{On Topic?}

    OnTopic -->|Yes| Target[Send to Target Image Model]
    OnTopic -->|No| Feedback

    Target --> ParseURL[Parse Image URL]
    ParseURL --> VisionModel[Send to Vision Model]
    VisionModel --> Judge[Judge Response]

    Judge --> CheckBest{Best Score?}

    CheckBest -->|Yes| UpdateBest[Update Best Response]
    CheckBest -->|No| Feedback

    UpdateBest --> Feedback[Prepare Feedback]
    Feedback --> CheckEnd{End Conditions:
1. Score ≥ 10
2. Max Iterations
Reached}

    CheckEnd -->|No| Loop
    CheckEnd -->|Yes| End([End])

    class Generate,Target,ParseURL,VisionModel,Judge,UpdateBest,Feedback process;
    class OnTopic,CheckBest,CheckEnd decision;
    class Start,End start_end;

Unblocking Feature

Overview

The unblocking feature helps red team providers handle situations where a target model asks clarifying questions that block the conversation from progressing. When enabled, it automatically detects and responds to these blocking questions to keep the adversarial conversation flowing.

How It Works

During a red team attack (e.g., Crescendo, GOAT, Custom providers), the unblocking system:

1. Analyzes the target model's response to detect if it's asking a blocking question
2. If a blocking question is detected, generates an appropriate answer
3. Sends the answer to continue the conversation
4. Continues with the red team attack

Example blocking questions:

• "What specific industry are you in?"
• "Can you provide more details about your use case?"
• "Which country are you located in?"

When to Enable

Enable unblocking (PROMPTFOO_ENABLE_UNBLOCKING=true) when:

• Testing conversational agents that frequently ask clarifying questions
• Running red team evaluations against customer service bots or domain-specific assistants
• You want more realistic multi-turn adversarial conversations
• The target system requires contextual information to proceed

Keep disabled (default) when:

• Testing simple question-answering systems
• You want faster red team evaluations (fewer API calls)
• You want to measure how well the target handles ambiguous queries
• Budget constraints are a concern (each unblocking check adds cost)

Configuration

Set the environment variable before running red team evaluations:

# Enable unblocking
export PROMPTFOO_ENABLE_UNBLOCKING=true

# Run red team eval
promptfoo eval

Tradeoffs

Pros:

• More realistic adversarial testing
• Better coverage for conversational systems
• Helps surface vulnerabilities in multi-turn scenarios

Cons:

• Increases evaluation time and cost
• Adds extra API calls to the remote inference service
• May generate unnecessary responses if the target model isn't actually blocking

Implementation Details

The unblocking feature is implemented in shared.ts and uses:

• Remote inference for blocking question detection
• The blocking-question-analysis task
• Conversation history and goal context for generating appropriate answers

Unblocking is available in:

• Crescendo Provider
• GOAT Provider
• Custom Provider

References

• Based on research from: Jailbroken: How Does LLM Safety Training Fail?
• Based on research from: A STRONGREJECT for Empty Jailbreaks