ContextQMD
Back to Promptfoo

amazon-bedrock/agents (AWS Bedrock Agents Example)

examples/amazon-bedrock/agents/README.md

0.121.95.3 KB
Original Source

amazon-bedrock/agents (AWS Bedrock Agents Example)

This example demonstrates how to use AWS Bedrock Agents with promptfoo to test and evaluate deployed AI agents, including both single-agent and multi-agent scenarios.

You can run this example with:

bash
npx promptfoo@latest init --example amazon-bedrock/agents
cd amazon-bedrock/agents

Prerequisites

  1. An AWS account with Bedrock Agents access

  2. One or more deployed Bedrock agents (get agent IDs from the AWS Console)

  3. AWS credentials configured (via environment variables, AWS CLI, or IAM role)

  4. Install the required AWS SDK:

    bash
    npm install @aws-sdk/client-bedrock-agent-runtime

Setup

  1. Get your Agent ID(s):

    • Go to the AWS Bedrock Console
    • Navigate to Agents
    • Copy your agent ID(s) (format: ABCDEFGHIJ)

  2. Configure AWS Credentials (choose one method):

    Via environment variables:

    bash
    export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_REGION=us-east-1

    Via AWS CLI profile:

    bash
    aws configure --profile my-bedrock-profile

    Via IAM role (if running on EC2/Lambda)

  3. Choose your configuration:

    • promptfooconfig.yaml: Basic single-agent configuration
    • promptfooconfig.multi-agent.yaml: Advanced multi-agent system configuration

Running the Examples

Single Agent Example

bash
# Run basic agent evaluation
npx promptfoo eval -c promptfooconfig.yaml

# View results in the web UI
npx promptfoo view

Multi-Agent Example

bash
# Run multi-agent system evaluation
npx promptfoo eval -c promptfooconfig.multi-agent.yaml

# View results in the web UI
npx promptfoo view

Configuration Options

Basic Usage

yaml
providers:
  - bedrock-agent:YOUR_AGENT_ID

Advanced Single Agent Configuration

yaml
providers:
  - id: bedrock-agent:my-agent
    config:
      agentId: YOUR_AGENT_ID
      agentAliasId: PROD_ALIAS # Optional: specific version/alias
      region: us-east-1 # AWS region
      sessionId: session-123 # Maintain conversation state
      enableTrace: true # Get detailed execution traces
      memoryId: SHORT_TERM_MEMORY # or LONG_TERM_MEMORY

Multi-Agent System Configuration

yaml
providers:
  # Technical Support Agent
  - id: tech-agent
    provider: bedrock-agent:TECH_AGENT_ID
    config:
      agentId: TECH_AGENT_ID
      agentAliasId: TECH_ALIAS_ID
      region: us-east-1
      enableTrace: true
      memoryId: LONG_TERM_MEMORY

  # Billing Agent
  - id: billing-agent
    provider: bedrock-agent:BILLING_AGENT_ID
    config:
      agentId: BILLING_AGENT_ID
      agentAliasId: BILLING_ALIAS_ID
      region: us-east-1
      enableTrace: true

Features

Session Management

The provider supports maintaining conversation state across multiple interactions:

yaml
config:
  sessionId: my-session-123 # Use the same session ID for related queries

Memory Integration

Enable agent memory for context-aware responses:

yaml
config:
  memoryId: LONG_TERM_MEMORY # or SHORT_TERM_MEMORY

Trace Information

Get detailed execution traces including tool calls and reasoning:

yaml
config:
  enableTrace: true # Response will include trace metadata

Testing Scenarios

Single Agent Tests

The basic config includes tests for:

  • Basic agent responses
  • Tool/function calling (e.g., calculator)
  • Memory retention
  • Multi-turn conversations

Multi-Agent System Tests

The multi-agent config includes tests for:

  • Specialized agent capabilities (technical, billing, product)
  • Cross-functional issue handling
  • Agent collaboration and coordination
  • Escalation management
  • Performance and latency validation

Multi-Agent System Architecture

The multi-agent example demonstrates a customer support system with specialized agents:

text
Customer Query
     ↓
[Supervisor Agent] ← Monitors & Routes
     ↓
┌─────────────────┬─────────────────┬──────────────────┐
│  Tech Agent     │  Billing Agent  │  Product Agent   │
│  (Technical)    │  (Payments)     │  (Recommendations)│
└─────────────────┴─────────────────┴──────────────────┘

Troubleshooting

  1. Authentication Error: Ensure AWS credentials are properly configured
  2. Agent Not Found: Verify the agent ID and region
  3. Permissions Error: Check IAM permissions for bedrock:InvokeAgent
  4. Timeout: Large agent responses may take time; adjust timeout if needed
  5. Multi-Agent Issues: Ensure all agent IDs and aliases are correct in the config

IAM Permissions

Your AWS credentials need the following permissions:

json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["bedrock:InvokeAgent"],
      "Resource": "arn:aws:bedrock:*:*:agent/*"
    }
  ]
}

Learn More