Smoke Tests Plan

Comprehensive smoke test plan for the promptfoo CLI and library. This document serves as the checklist and specification for smoke tests that verify the built package works correctly across critical user flows.

Test location: test/smoke/ Run command: npm run test:smoke

Philosophy

1. Test the built package (dist/), not source code
2. No external API dependencies - use echo provider, local scripts, or mock servers
3. Fast execution - target < 3 minutes total
4. Cover critical paths - CLI commands, config loading, provider initialization

Running Smoke Tests

# Run all smoke tests
npm run test:smoke

# Run individual test suites
npm run test:smoke:cli      # CLI binary tests
npm run test:smoke:eval     # Eval pipeline tests

Test Location

All smoke tests live in test/smoke/:

test/smoke/
├── cli.test.ts              # CLI command tests
├── eval.test.ts             # Eval pipeline tests
├── providers.test.ts        # Provider loading tests
├── configs.test.ts          # Config format tests
├── data-loading.test.ts     # Data source tests
├── fixtures/
│   ├── configs/             # Config format examples
│   ├── providers/           # Echo-based providers
│   ├── data/                # Test data files
│   └── assertions/          # Assertion scripts
└── scripts/
    └── run-all.sh           # Shell-based smoke tests

Test Checklist

1. CLI Binary Tests

1.1 Basic CLI Operations

1.2 Init Command

1.3 Validate Command

1.4 Eval Command (Core)

1.5 List/Show/Export Commands

1.6 Cache Commands

1.7 Exit Codes

1.8 Filter Flags

Test count and content filters that control which tests run.

1.8.1 Count-Based Filters

1.8.2 Pattern Filters

1.8.3 Metadata Filters

1.8.4 Provider Filters

1.8.5 History-Based Filters

1.8.6 Combined Filters

1.9 Variable and Prompt Flags

Flags that modify variables or prompt content.

1.10 Execution Control Flags

Flags that control how tests execute.

1.11 Output and Metadata Flags

Flags that affect output and eval metadata.

1.12 Resume and Retry Flags

Flags for resuming or retrying evaluations.

2. Config Format Tests

2.1 YAML Configs

2.2 JSON Configs

2.3 JavaScript Configs

2.4 TypeScript Configs

3. Provider Tests

3.1 Built-in Providers (No API Key)

3.2 JavaScript Providers

3.3 TypeScript Providers

3.4 Python Providers

3.5 Ruby Providers

3.6 Go Providers

3.7 HTTP Providers

3.8 HTTP Auth Configurations

4. Data Loading Tests

4.1 Vars Loading

4.2 Tests Loading

4.3 Prompts Loading

5. Assertion Tests

5.1 Built-in Assertions

5.2 Script Assertions

5.3 Model-Graded Assertions (Config Validation Only)

6. Transform Tests

6.1 Response Transforms

6.2 Prompt Transforms

7. Feature Integration Tests

7.1 Provider Config Options

7.2 DefaultTest

7.3 Scenarios

8. Example Config Smoke Tests

Validate existing examples work with echo provider substitution:

Implementation Priority

Phase 1: Foundation (Must Ship) - COMPLETE

• 1.1.1-1.1.5: Basic CLI operations
• 1.4.1-1.4.7: Basic eval with echo (output formats, flags)
• 1.7.1-1.7.3: Exit codes
• 2.1.1: YAML config
• 3.1.1: Echo provider
• 4.2.1: Inline tests
• 1.2.1: Init command
• 1.3.1-1.3.2: Validate command
• 1.5.1-1.5.2: List commands
• 1.6.1: Cache commands

Phase 2: Config & Provider Formats - PARTIAL

• 2.2.1: JSON config
• 2.3.1: CJS config with module.exports
• 2.3.5: ESM config with export default
• 2.3.2-2.3.4, 2.3.6-2.3.7: Other JS config variants
• 2.4.1: TypeScript config with export default
• 2.4.2-2.4.6: Other TS config variants
• 3.2.1: CJS provider class
• 3.2.4: ESM provider class
• 3.2.2-3.2.3, 3.2.5: Other JS provider variants
• 3.3.1: TypeScript provider class
• 3.3.2-3.3.3: Other TS provider variants

Phase 3: Script Providers - PARTIAL

• 3.4.1: Python provider (default call_api)
• 3.4.2: Python provider named function
• 3.4.3-3.4.7: Other Python provider variants
• 3.5.1-3.5.3: Ruby provider variants
• 3.6.1-3.6.2: Go provider variants
• 3.1.2: Exec provider

Phase 4: Data & Assertions - PARTIAL

• 4.2.2: CSV file tests
• 4.2.3: JSON file tests
• 4.2.4: JSONL file tests
• 4.2.5: YAML file tests
• 4.2.7: JS test generator
• 4.3.2: File ref prompts
• 4.2.6, 4.2.8-4.2.10: Other data loading formats (XLSX, TS/Python generators, glob)
• 5.1.1: Contains assertion
• 5.1.2: Not-contains assertion
• 5.1.3: Equals assertion
• 5.1.4: Starts-with assertion
• 5.1.5: Regex assertion
• 5.1.6: Is-JSON assertion
• 5.1.7: Contains-json assertion
• 5.2.1: Inline JavaScript assertion
• 5.2.2: JavaScript file assertion
• 5.2.5: Python file assertion
• 5.1.8-5.1.11: Other built-in assertions (json-schema, cost, latency, perplexity)
• 5.2.3-5.2.4, 5.2.6: Other script assertion variants

Phase 5: HTTP & Auth

• 3.7.1-3.7.5: HTTP provider basics
• 3.8.1-3.8.10: All auth configurations

Phase 6: Filter & Flag Tests - PARTIAL

High-value tests for CLI filter flags and execution options.

Priority 1 - Count/Pattern Filters:

• 1.8.1.1: First N tests (--filter-first-n 2)
• 1.8.1.2: First N > total (returns all)
• 1.8.1.4: Sample N tests (--filter-sample 2)
• 1.8.2.1: Pattern filter (--filter-pattern "user.*test")
• 1.8.2.3: Pattern no match (0 tests)
• 1.8.3.1: Metadata filter (--filter-metadata category=auth)
• 1.8.3.2: Metadata partial match
• 1.8.3.3: Metadata array match

Priority 2 - Provider Filters:

• 1.8.4.1: Provider by ID (--filter-providers "echo")
• 1.8.4.2: Provider by label regex
• 1.8.6.1: Combined filters (pattern + first-n)

Priority 3 - Variable/Prompt Flags:

• 1.9.1: Single var (--var name=Alice)
• 1.9.2: Multiple vars
• 1.9.3: Var precedence (test vars override --var)
• 1.9.5: Prompt prefix
• 1.9.6: Prompt suffix
• 1.9.7: Prefix + suffix combined

Priority 4 - Output/Execution Flags:

• 1.10.1: Delay between tests (--delay 100)
• 1.10.5: No table output
• 1.11.1: Description flag
• 1.11.2: Multiple output files
• 1.11.3: No write flag

Priority 5 - History-Based Filters (more complex):

• 1.8.5.1: Filter failing from file
• 1.8.5.3: Filter errors only from file
• 1.12.1: Resume evaluation
• 1.12.3: Retry errors

Phase 7: Integration & Polish - PARTIAL

• 6.1.1: Transform response expression
• 6.1.2-6.1.3: Other transform variants
• 7.1.1: Provider with config options
• 7.1.2-7.1.3: Other provider config variants
• 7.2.1: DefaultTest feature
• 7.2.2: File defaultTest
• 7.3.1: Scenarios feature
• 8.1-8.10: Example config tests

Phase 8: Advanced Features - PARTIAL

Advanced CLI features and assertion capabilities.

• 1.4.8: Environment file loading (--env-file)
• 1.4.4b: HTML output format
• 4.3.1b: Multiple prompts (A/B testing)
• 4.3.2b: Multiple file prompts (file:// references)
• 5.1.2b: icontains assertion (case-insensitive)
• 5.1.5b: Regex end-of-string pattern ($ anchor)
• 5.3.1: Assertion weights
• 7.4.1: Test threshold option (partial assertion passes)
• 1.4.9: --grader flag for model-graded assertions
• 1.12.1: Resume evaluation (--resume)
• 1.12.3: Retry errors (--retry-errors)
• 2.5.1: Config extends feature

Cross-Platform Testing

OS Matrix

Script Language Matrix

CI Integration

Add to .github/workflows/main.yml:

smoke-tests:
  name: Smoke Tests
  runs-on: ubuntu-latest
  needs: [build]
  steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: '20'
    - uses: actions/setup-python@v5
      with:
        python-version: '3.11'

    - name: Install dependencies
      run: npm ci

    - name: Build
      run: npm run build

    - name: Smoke Tests
      run: npm run test:smoke

Writing New Smoke Tests

Guidelines

1. Use echo provider - No external API dependencies
2. Keep fixtures minimal - Only what's needed to test the feature
3. Test one thing - Each test should verify a single capability
4. Include negative tests - Verify errors are handled correctly
5. Document the test - Add entry to checklist above

Echo-Based Provider Template

# test/smoke/fixtures/providers/echo-provider.py
def call_api(prompt, options, context):
    """Echo provider that returns the prompt back."""
    return {
        "output": f"Echo: {prompt}",
        "tokenUsage": {"total": len(prompt), "prompt": len(prompt), "completion": 0}
    }

// test/smoke/fixtures/providers/echo-provider.js
module.exports = class EchoProvider {
  constructor(options) {
    this.id = options.id || 'echo-js';
  }
  id() {
    return this.id;
  }
  async callApi(prompt) {
    return { output: `Echo: ${prompt}` };
  }
};

Config Fixture Template

# test/smoke/fixtures/configs/basic.yaml
description: 'Smoke test - basic eval'
providers:
  - echo
prompts:
  - 'Hello {{name}}'
tests:
  - vars:
      name: World
    assert:
      - type: contains
        value: Hello
      - type: contains
        value: World

Key Findings

Important discoveries made during smoke test implementation that may inform future development:

Provider Named Function Syntax

The :functionName syntax (e.g., file://provider.py:custom_fn) is only supported for:

• Python providers - file://provider.py:custom_fn works
• Ruby providers - file://provider.rb:custom_fn works
• Go providers - file://main.go:CustomFn works

It is NOT supported for JavaScript/TypeScript providers. JS/TS providers must export a class:

// Correct - class export
module.exports = class MyProvider {
  async callApi(prompt) {
    return { output: prompt };
  }
};

// NOT supported - named function export for providers
// module.exports.callApi = async (prompt) => { ... };

The :functionName syntax IS supported for JavaScript in other contexts:

• Assertions: type: javascript, value: file://assert.js:checkFn
• Transforms: transform: file://transform.js:transformFn
• Test generators: tests: file://tests.js:generateTests

Assertion Behavior

contains-json Assertion

The contains-json assertion behaves differently based on whether a value is provided:

• Without value: Checks that the output contains valid JSON somewhere
• With value: Validates extracted JSON against a JSON Schema (not a subset match)

# Just check for valid JSON presence
- type: contains-json

# Validate against JSON Schema
- type: contains-json
  value:
    type: object
    required: [status, code]
    properties:
      status: { type: string }
      code: { type: number }

Scenarios Configuration

The scenarios[].config field expects an array of variable configurations, not a plain object:

# Correct
scenarios:
  - config:
      - vars:
          region: US
      - vars:
          region: EU
    tests:
      - vars: { name: Alice }

# Incorrect - will fail validation
scenarios:
  - config:
      region: US  # This is wrong
    tests:
      - vars: { name: Alice }

Exit Codes

The CLI uses specific exit codes:

Output JSON Structure

When exporting results to JSON (-o output.json), key paths:

• results.prompts[].provider - Provider label/ID for each prompt
• results.results[].success - Boolean indicating if all assertions passed
• results.results[].response.output - The LLM output text
• results.results[].provider.label - Provider label if configured
• results.results[].gradingResult.componentResults[] - Individual assertion results

Echo Provider Behavior

The built-in echo provider returns the prompt exactly as-is. This is useful for:

• Testing assertion logic without API calls
• Verifying prompt template rendering
• Testing data loading and variable substitution

To test JSON-related assertions, include JSON in the prompt itself:

prompts:
  - 'Response: {"status": "ok", "count": 42}'
tests:
  - assert:
      - type: contains-json

Test Isolation

Each smoke test file creates its own temporary output directory and cleans it up in afterAll. This ensures tests don't interfere with each other when run in parallel.

const OUTPUT_DIR = path.resolve(__dirname, '.temp-output-unique-name');

beforeAll(() => {
  fs.mkdirSync(OUTPUT_DIR, { recursive: true });
});

afterAll(() => {
  fs.rmSync(OUTPUT_DIR, { recursive: true, force: true });
});

Available Assertion Types

There is no ends-with assertion type. To check string suffixes, use the regex assertion with a $ anchor:

# Check if output ends with "42."
- type: regex
  value: '42\.$'

Available assertion types include: contains, contains-all, contains-any, icontains, icontains-all, icontains-any, equals, starts-with, regex, is-json, contains-json, is-html, is-xml, is-sql, javascript, python, and model-graded assertions.

Glob Patterns for Prompts

Glob patterns in prompts (e.g., prompts/*.txt) have path resolution issues when used with file:// prefix. Use explicit file references instead:

# Works - explicit file references
prompts:
  - file://../prompts/greeting.txt
  - file://../prompts/farewell.txt

# Has issues - glob pattern
prompts:
  - file://prompts/*.txt

Multiple Config Files Behavior

Using multiple -c flags doesn't deeply merge configs. Each config is processed separately with defaults for missing properties. The second config will get a default {{prompt}} prompt if prompts aren't specified.

To compose configs, use explicit file:// references within a single config or specify all required properties in each config file.

HTML Output Format

The HTML output uses lowercase <!doctype html> (valid HTML5) rather than uppercase <!DOCTYPE html>.

Bug Regression Tests (0.120.x)

Critical bugs identified in versions 0.120.0-0.120.3 that should have smoke test coverage. These bugs represent real issues users encountered after the major ESM migration.

10. ESM Migration Bugs (0.120.0)

The 0.120.0 release migrated from CommonJS to ESM, causing several regressions:

10.1 Module Loading

10.2 Provider Path Resolution

10.3 Cache & Config

10.4 CLI Issues

10.5 Language Providers

11. Version 0.120.1-0.120.2 Bugs

11.1 Database & Migrations

11.2 Parsing Issues

11.3 Assertion Improvements

11.4 HTTP Provider

12. Key Regression Test Implementations

Priority smoke tests to add based on critical 0.120.x bugs:

12.1 Module Loading Regression Tests

# Test 10.1.1: CJS provider with module.exports still works
# Fixture: test/smoke/fixtures/providers/cjs-module-exports.js
providers:
  - file://providers/cjs-module-exports.js

# Test 10.1.3: Inline JS with process.mainModule (requires Node.js CJS compat)
tests:
  - assert:
      - type: javascript
        value: |
          // This should not throw even though process.mainModule is undefined in ESM
          const output = context.output || '';
          return output.includes('test');

12.2 Provider Path Resolution Tests

# Test 10.2.1: Provider path relative to config file, NOT cwd
# Config at: test/smoke/fixtures/subdir/config-relative-provider.yaml
# Provider at: test/smoke/fixtures/subdir/local-provider.js
# Run from: test/smoke/ (different directory than config)
providers:
  - file://./local-provider.js # Should resolve relative to config, not cwd

12.3 Config Option Tests

# Test 10.3.2: maxConcurrency in config.yaml is respected
defaultTest:
  options:
    maxConcurrency: 1
providers:
  - echo
prompts:
  - 'Test {{n}}'
tests:
  - vars: { n: 1 }
  - vars: { n: 2 }
  - vars: { n: 3 }
# Verify tests run sequentially (timing check)

Implementation Priority for Regression Tests

Phase 1: High Priority (Add Now)

• 10.1.1: CJS module.exports provider loading
• 10.2.1: Provider path resolution from config directory
• 10.3.2: maxConcurrency in config file
• 11.2.1: JSON chat message parsing

Phase 2: Medium Priority

• 10.1.3: Inline JS with process.mainModule shim
• 10.5.1: Go provider basic test
• 10.5.2: Ruby provider basic test

Phase 3: Lower Priority (Complex Setup)

• 10.2.2: Python wrapper path from different CWD
• 11.4.1: HTTP provider complex body parsing

Implemented Tests Summary

Current smoke test coverage: