ContextQMD
Back to Promptfoo

Layer Strategy

site/docs/red-team/strategies/layer.md

0.121.913.0 KB
Original Source

Layer Strategy

The Layer strategy allows you to compose multiple red team strategies sequentially, creating sophisticated attack chains by feeding the output of one strategy into the next. This enables complex transformations like applying multiple encoding layers or combining agentic attacks with multi-modal output (audio/image).

Quick Start

Apply multiple strategies in sequence:

yaml
redteam:
  strategies:
    - id: layer
      config:
        steps:
          - base64 # First encode as base64
          - rot13 # Then apply ROT13

How It Works

The Layer strategy operates in two distinct modes depending on the types of steps you include:

Mode 1: Transform Chain (No Agentic Steps)

When all steps are transforms (base64, rot13, leetspeak, etc.), layer works as a simple pipeline:

  1. Sequential Processing: Each step receives the output from the previous step
  2. Cumulative Transformation: Changes stack (e.g., text → base64 → rot13)
  3. Final Output Only: Only the last step's outputs become test cases
  4. Pre-Evaluation: All transforms are applied before sending to the target

Mode 2: Agentic + Per-Turn Transforms

When the first step is an agentic strategy (hydra, crescendo, goat, jailbreak, etc.), layer enables powerful multi-turn/multi-attempt attacks with per-turn transformations:

  1. Agentic Orchestration: The agentic strategy controls the attack loop
  2. Per-Turn Transforms: Remaining steps (e.g., audio, image) are applied to each turn dynamically
  3. Multi-Modal Attacks: Combine conversation-based attacks with audio/image delivery
yaml
providers:
  - id: openai:chat:gpt-4o-audio-preview
    config:
      modalities: ['text', 'audio']
      audio:
        voice: 'alloy'
        format: 'mp3'

prompts:
  - |
    [{"role": "user", "content": [{"type": "input_audio", "input_audio": {"data": "{{prompt}}", "format": "mp3"}}]}]

redteam:
  purpose: 'A helpful customer service assistant'
  plugins:
    - harmful:hate
  strategies:
    - id: layer
      config:
        steps:
          - id: jailbreak
            config:
              maxIterations: 2
          - audio

Ordering Rules

Agentic strategies (hydra, crescendo, goat, jailbreak) must come first (max 1), and multi-modal strategies (audio, image) must come last (max 1). Text transforms (base64, rot13, leetspeak) can be chained in between.

Valid Patterns

yaml
# Transform chain (no agentic)
steps: [base64, rot13]

# Transform + multi-modal
steps: [leetspeak, audio]

# Agentic only
steps: [jailbreak:hydra]

# Agentic + multi-modal (recommended for voice/vision targets)
steps: [jailbreak:hydra, audio]

Invalid Patterns

yaml
# ❌ Wrong: Agentic not first (transforms will corrupt the goal)
steps: [base64, jailbreak:hydra]

# ❌ Wrong: Multi-modal not last
steps: [audio, base64]

# ❌ Wrong: Multiple agentic strategies
steps: [hydra, crescendo]

# ❌ Wrong: Multiple multi-modal strategies
steps: [audio, image]

:::warning

Transforms before an agentic strategy modify the attack goal, not each turn—rarely useful.

:::

Configuration Options

Label (for Multiple Layer Strategies)

Use the label field to differentiate multiple layer strategies in the same config:

yaml
redteam:
  strategies:
    # Multiple layer strategies with unique labels
    - id: layer
      config:
        label: hydra-audio # Unique identifier
        steps:
          - jailbreak:hydra
          - audio

    - id: layer
      config:
        label: crescendo-audio # Different label = different strategy
        steps:
          - crescendo
          - audio

Without labels, layer strategies are deduplicated based on their steps. With labels, each labeled layer is treated as a distinct strategy.

Basic Configuration

Simple string-based steps for built-in strategies:

yaml
redteam:
  strategies:
    - id: layer
      config:
        steps:
          - base64 # Simple strategy reference
          - leetspeak # Another simple strategy
          - rot13 # Final transformation

Advanced Configuration

Object-based steps with individual configurations:

yaml
redteam:
  strategies:
    - id: layer
      config:
        steps:
          # Step with custom configuration
          - id: jailbreak
            config:
              maxIterations: 10

          # Simple step
          - hex

          # Custom strategy script
          - id: file://custom-obfuscator.js
            config:
              intensity: high

Step-Level Plugin Targeting

Control which plugins each step applies to:

yaml
redteam:
  strategies:
    - id: layer
      config:
        plugins: ['harmful', 'pii'] # Default for all steps
        steps:
          # Override for specific step
          - id: jailbreak
            config:
              plugins: ['harmful'] # Only apply to harmful plugin

          # Uses default plugins from parent config
          - rot13

          # Different plugin set
          - id: base64
            config:
              plugins: ['pii', 'contracts']

Example Scenarios

Multi-Turn Audio Attack

Test voice-enabled AI agents with sophisticated jailbreak attempts:

yaml
redteam:
  strategies:
    - id: layer
      config:
        steps:
          - jailbreak:hydra # Multi-turn jailbreak
          - audio # Convert each turn to speech

Hydra will orchestrate the attack, and each turn's prompt will be converted to audio before being sent to your target. Your custom provider receives a hybrid payload with conversation history (as text) and the current turn (as audio).

Multi-Turn Image Attack

Test vision-enabled AI agents:

yaml
redteam:
  strategies:
    - id: layer
      config:
        steps:
          - crescendo # Gradual escalation attack
          - image # Convert each turn to image

Single-Turn Audio Attack

For multi-attempt strategies (jailbreak, jailbreak:meta, jailbreak:tree), each independent attempt is converted to audio:

yaml
redteam:
  strategies:
    - id: layer
      config:
        steps:
          - jailbreak:meta # Meta-agent generates attack variants
          - audio # Each variant converted to audio

Double Encoding Attack

Apply multiple encoding layers to evade detection:

yaml
redteam:
  strategies:
    - id: layer
      config:
        steps:
          - hex # First encode as hexadecimal
          - base64 # Then base64 encode

Progressive Obfuscation

Stack multiple encoding techniques for maximum obfuscation:

yaml
redteam:
  strategies:
    - id: layer
      config:
        steps:
          - leetspeak # First apply leetspeak
          - hex # Then hex encode
          - base64 # Finally base64 encode

Injection Chain

Combine prompt injection with encoding:

yaml
redteam:
  strategies:
    - id: layer
      config:
        steps:
          - prompt-injection # Add injection payloads
          - rot13 # Obfuscate the injection

Custom Strategy Pipeline

Use custom scripts in your pipeline:

yaml
redteam:
  strategies:
    - id: layer
      config:
        steps:
          - file://strategies/add-context.js
          - base64
          - file://strategies/final-transform.js

Custom Provider for Audio/Image Targets

When using layer with audio or image transforms, your custom provider receives a hybrid JSON payload. Here's how to handle it:

javascript
class AudioProvider {
  id() {
    return 'audio-target';
  }

  async callApi(prompt) {
    let messages = [];

    // Check for hybrid payload from layer strategy
    if (typeof prompt === 'string' && prompt.startsWith('{')) {
      try {
        const parsed = JSON.parse(prompt);
        if (parsed._promptfoo_audio_hybrid) {
          // Build messages from conversation history (text)
          messages = (parsed.history || []).map((msg) => ({
            role: msg.role,
            content: msg.content,
          }));

          // Add current turn with audio/image
          const currentTurn = parsed.currentTurn;
          if (currentTurn?.audio?.data) {
            messages.push({
              role: currentTurn.role,
              content: [
                {
                  type: 'input_audio',
                  input_audio: {
                    data: currentTurn.audio.data,
                    format: currentTurn.audio.format || 'mp3',
                  },
                },
              ],
            });
          } else if (currentTurn?.image?.data) {
            messages.push({
              role: currentTurn.role,
              content: [
                {
                  type: 'image_url',
                  image_url: {
                    url: `data:image/${currentTurn.image.format || 'png'};base64,${currentTurn.image.data}`,
                  },
                },
              ],
            });
          }
        }
      } catch (e) {
        // Fallback to treating as plain text
        messages = [{ role: 'user', content: prompt }];
      }
    }

    // Call your audio-capable API (e.g., OpenAI gpt-4o-audio-preview)
    const response = await yourApiCall(messages);

    return {
      output: response.text,
      audio: response.audio
        ? {
            data: response.audio.data,
            transcript: response.audio.transcript,
            format: 'mp3',
          }
        : undefined,
    };
  }
}

module.exports = AudioProvider;

The hybrid payload structure:

json
{
  "_promptfoo_audio_hybrid": true,
  "history": [
    { "role": "user", "content": "Hello" },
    { "role": "assistant", "content": "Hi there!" }
  ],
  "currentTurn": {
    "role": "user",
    "transcript": "Tell me about...",
    "audio": { "data": "base64...", "format": "mp3" }
  }
}

Performance Considerations

Test Case Multiplication

Be aware of test case growth when combining strategies:

  • Some strategies may multiply test cases (e.g., if you set global language: ['en', 'es', 'fr'], all test cases multiply by 3)
  • Layered strategies process sequentially, so test count growth is usually linear
  • Plan your test counts accordingly to avoid excessive evaluation time

Optimization Tips

  1. Order Matters: Place filtering strategies early, expanding strategies late
  2. Avoid Redundancy: Don't use the same strategy both in a layer and at the top level
  3. Test Small First: Validate with a subset before scaling up
  4. Monitor Memory: Complex layers with many steps can consume significant memory

Implementation Notes

  • Empty Step Handling: If any step returns no test cases, subsequent steps receive an empty array
  • Error Handling: Failed steps are logged and skipped; the pipeline continues
  • Metadata Preservation: Each step preserves and extends test case metadata
  • Strategy Resolution: Supports both built-in strategies and file:// custom strategies

Tips and Best Practices

  1. Agentic First: When combining agentic with transforms, always place the agentic strategy first
  2. Multi-Modal Last: Audio and image transforms must be the final step
  3. Logical Combinations: Stack strategies that make semantic sense together
  4. Debug Incrementally: Test each step individually before combining
  5. Document Complex Layers: Add comments explaining the attack chain logic
  6. Consider Target Models: Voice/vision targets need appropriate multi-modal steps
  7. Use Plugin Targeting: Focus different steps on different vulnerability types
  8. Custom Provider Required: Audio/image attacks require a custom provider that handles the hybrid payload format

Supported Agentic Strategies

The following strategies can be used as the first step with per-turn transforms:

StrategyTypeDescription
jailbreak:hydraMulti-turnBranching conversation attack
crescendoMulti-turnGradual escalation attack
goatMulti-turnGoal-oriented adversarial testing
customMulti-turnCustom multi-turn strategy
jailbreakMulti-attemptIterative single-turn attempts
jailbreak:metaMulti-attemptMeta-agent attack generation
jailbreak:treeMulti-attemptTree-based attack search