ContextQMD
Back to Baml

Test

fern/03-reference/baml-cli/test.mdx

0.222.05.4 KB
Original Source

The test command runs BAML function tests defined in your BAML files. It provides comprehensive testing capabilities including filtering, parallel execution, and various output formats.

Usage

baml-cli test [OPTIONS]

Options

OptionDescriptionDefault
--from <PATH>Path to the baml_src directory./baml_src
--listOnly list selected tests without running themfalse
-i, --include <PATTERN>Include specific functions or tests (can be used multiple times)Run all tests
-x, --exclude <PATTERN>Exclude specific functions or tests (can be used multiple times)None
--parallel <NUM>Number of tests to run in parallel10
--pass-if-no-testsPass if no tests are selectedfalse
--require-human-evalFail if any tests need human evaluationtrue
--dotenvLoad environment variables from .env filetrue
--dotenv-path <PATH>Path to custom .env fileNone

Description

The test command performs the following actions:

  1. Discovers and parses all test cases defined in BAML files
  2. Applies include/exclude filters to select which tests to run
  3. Executes tests in parallel (configurable concurrency)
  4. Reports results with detailed output and assertions
  5. Supports various output formats and CI integration

Test Filtering

The --include and --exclude options support powerful pattern matching:

Pattern Syntax

  • * matches any characters within a name
  • FunctionName::TestName matches a specific test in a specific function
  • FunctionName:: matches all tests in a function
  • ::TestName matches a test name across all functions
  • Multiple patterns can be combined

Examples

bash
# Run all tests
baml-cli test

# List all available tests
baml-cli test --list

# Run tests for a specific function
baml-cli test -i "ExtractResume::"

# Run a specific test
baml-cli test -i "ExtractResume::TestBasicResume"

# Run all tests matching a pattern
baml-cli test -i "Extract*::"

# Run tests with multiple include patterns
baml-cli test -i "Extract*::" -i "Classify*::"

# Exclude specific tests
baml-cli test -x "ExtractResume::TestComplexResume"

# Combine include and exclude (exclude takes precedence)
baml-cli test -i "Extract*::" -x "*::TestSlow*"

Parallel Execution

Control the number of tests running simultaneously:

bash
# Run tests with default parallelism (10)
baml-cli test

# Run tests sequentially
baml-cli test --parallel 1

# Run with high parallelism
baml-cli test --parallel 20

Environment Variables

The test command automatically loads environment variables:

bash
# Use default .env file loading
baml-cli test

# Disable .env file loading
baml-cli test --no-dotenv

# Use custom .env file
baml-cli test --dotenv-path .env.test

Environment variables can also be set directly:

bash
# Set API keys for testing
OPENAI_API_KEY=... ANTHROPIC_API_KEY=... baml-cli test

Exit Codes

The test command returns different exit codes based on results:

Exit CodeMeaning
0All tests passed
1Test failures occurred
2Tests require human evaluation
3Test execution was cancelled
4No tests were found to run

Examples

Basic Testing

bash
# Run all tests in the project
baml-cli test

# Run tests from a specific directory
baml-cli test --from /path/to/my/baml_src

Development Workflow

bash
# Run tests for a function you're developing
baml-cli test -i "MyNewFunction::"

# Run specific test while debugging
baml-cli test -i "MyFunction::TestEdgeCase"

# List tests to see what's available
baml-cli test --list -i "Extract*::"

CI/CD Integration

bash
# Fail fast on first assertion failure
baml-cli test --require-human-eval

# Allow tests that need human evaluation to pass
baml-cli test --no-require-human-eval

# Run with controlled parallelism for CI
baml-cli test --parallel 5

Test Discovery

bash
# See all available tests
baml-cli test --list

# See tests for specific functions
baml-cli test --list -i "ClassifyMessage::"

# See what tests would run with filters
baml-cli test --list -i "Extract*::" -x "*::TestSlow*"

Test Definition

Tests are defined in BAML files using the test block syntax:

baml
function ExtractResume(resume: string) -> Resume {
  client GPT4
  prompt #"Extract resume information: {{resume}}"#
}

test TestBasicResume {
  functions [ExtractResume]
  args {
    resume "John Doe\[email protected]\nSoftware Engineer"
  }
  @@assert({{ this.name == "John Doe" }})
  @@assert({{ this.email == "[email protected]" }})
}
  • baml dev - Development server with hot reload for interactive testing
  • baml serve - Production server for HTTP API testing
  • baml generate - Generate client code before running tests