google-aistudio-gemini (Google AI Studio (Gemini and Gemma) Example)

You can run this example with:

npx promptfoo@latest init --example google-aistudio-gemini
cd google-aistudio-gemini

This example demonstrates using Google's Gemini and Gemma models with promptfoo to evaluate math puzzle-solving capabilities.

Prerequisites

• promptfoo CLI installed (npm install -g promptfoo or brew install promptfoo)
• Google AI Studio API key set as GOOGLE_API_KEY

Available Models

The example tests across multiple Gemini and Gemma models:

• Gemma 4 31B IT - Open model with strong reasoning, coding, and agentic capabilities
• Gemma 4 26B A4B IT - Smaller open Gemma 4 model for lower-latency reasoning and coding evals
• Gemini 3.7 Flash - Latest Flash model for coding, multimodal reasoning, and agentic workflows
• Gemini 3.6 Flash - Previous-generation Flash model for coding and multi-step tasks
• Gemini 3.5 Flash - Frontier Flash model with high-effort thinking
• Gemini 3.5 Flash-Lite - Low-latency model for high-volume agentic tasks
• Gemini 3.1 Pro - Frontier model with improved reasoning and multimodal understanding
• Gemini 3 Flash - Frontier Flash model with strong reasoning at lower latency
• Gemini 3.1 Flash-Lite - Low-latency, cost-efficient model for high-volume tasks
• Gemini 2.5 Pro - Stable model with strong reasoning, coding, and multimodal understanding; also used with structured JSON output and function calling
• Gemini 2.5 Flash - Stable Flash model with enhanced reasoning and thinking capabilities
• Gemini 2.5 Flash-Lite - Cost-efficient and fast 2.5 model, optimized for high-volume, latency-sensitive tasks
• gemini-embedding-001 - Embedding model used for similarity-based assertions

Gemini 3.7 Flash, 3.6 Flash, and 3.5 Flash-Lite use thinkingLevel instead of deprecated temperature, topP, or topK controls. Gemini 3.7 Flash does not support MINIMAL thinking.

System Instructions from File

This example also demonstrates how to load system instructions from an external file using the file:// prefix:

providers:
  - id: google:gemini-2.5-pro
    label: gemini-with-system-instruction-file
    config:
      systemInstruction: file://system-instruction.txt

The system-instruction.txt file contains reusable instructions that can be:

• Shared across multiple configurations
• Version controlled separately
• Used to manage complex or lengthy system prompts

Image Understanding Example

This example also includes an image understanding configuration (promptfooconfig.image.yaml) that demonstrates:

• Multimodal capabilities: Using Gemini models to analyze and compare images
• Image file handling: Loading images using the file:// prefix
• Visual comparison: Testing the model's ability to identify and compare different images

Images should be placed on separate lines in the prompt. The file:// prefix automatically handles loading and encoding the images for the Gemini API.

Running the Eval

1. Get a local copy of the configuration:

promptfoo init --example google-aistudio-gemini

2. Run the examples:

# Basic math puzzle evaluation across multiple Gemini and Gemma models
promptfoo eval

# Image understanding and comparison evaluation
promptfoo eval -c promptfooconfig.image.yaml

3. View the results:

promptfoo view