BAML Language Testing Guide

Test Suites

Workflow: Debugging a Failing Test

1. Identify the issue in baml_lsp2_actions_tests

cargo test --package baml_lsp2_actions_tests

Look for errors in crates/baml_lsp2_actions_tests/test_files/syntax/.

2. Create a minimal repro in baml_tests

Create a new project directory:

mkdir -p crates/baml_tests/projects/my_repro/

Add a .baml file with the minimal repro case:

# crates/baml_tests/projects/my_repro/repro.baml

3. Run and generate snapshots

cargo test --package baml_tests my_repro

Accept new snapshots:

cargo insta accept --all

4. Inspect the output

Snapshots are created in crates/baml_tests/snapshots/my_repro/:

5. Fix the issue

Edit the relevant crate (baml_compiler_parser, baml_compiler_syntax, baml_compiler2_hir, etc.).

6. Re-run and update snapshots

# Update baml_tests snapshots
cargo test --package baml_tests my_repro
cargo insta accept --all

# Update baml_lsp2_actions_tests inline expectations
UPDATE_EXPECT=1 cargo test --package baml_lsp2_actions_tests

7. Verify all tests pass

# Library unit tests — always run these when Rust code changes
cargo test --lib

# Run all tests (can skip slow parser_stress with --skip parser_stress)
cargo test --package baml_tests -- --skip parser_stress
cargo test --package baml_lsp2_actions_tests

Quick Commands

# Run specific test project
cargo test --package baml_tests my_project_name

# Run all snapshot tests
cargo test --package baml_tests

# Run all snapshot tests (skip slow parser_stress tests)
cargo test --package baml_tests -- --skip parser_stress

# Run LSP tests and auto-update expectations
UPDATE_EXPECT=1 cargo test --package baml_lsp2_actions_tests

# Accept all pending snapshots
cargo insta accept --all

# Review snapshots interactively
cargo insta review

Key Files

• Lexer: crates/baml_compiler_lexer/src/tokens.rs
• Parser: crates/baml_compiler_parser/src/parser.rs
• Syntax kinds: crates/baml_compiler_syntax/src/syntax_kind.rs
• AST helpers: crates/baml_compiler_syntax/src/ast.rs
• HIR lowering: crates/baml_compiler2_hir/src/body.rs
• Type checking: crates/baml_compiler2_tir/src/builder.rs

DO NOT EDIT the diagnostics manually in baml_lsp2_actions_tests. Use UPDATE_EXPECT=1

Find the base-case that makes syntax fail and add that to baml_test with a good name and good folder organization.

A good place to start when given a diagnostic failure or parser issue is to create a focused compiler test and inspect its diagnostics and IR snapshots.

BEFORE you run these lsp tests with UPDATE_EXPECT, make sure to just run without it and figure out if the new results are what you expect.

Just because the existing file may say 'no diagnostics expected' doesn't mean it is correct by the way. We haven't finished implementing all diagnostics. You have to see if we added some other comments elsewhere in the file to see what we should sort of expect, or just inspect the behavior manually.

END-TO-END TESTING (running real BAML programs)

The snapshot/LSP suites above test the compiler internals. This section is about testing BAML end-to-end as a user would: write a .baml program, compile it, run it, and run its test blocks — using the real CLI. Use this when you want to know "does this actually work when someone writes it", not "what CST does the parser produce".

The binary

Build and use the local dev CLI (do not use a brew-installed baml — you must test this checkout):

cargo build -p baml_cli           # produces target/debug/baml-cli
BAML=/Users/aaron/projects/baml/baml_language/target/debug/baml-cli

It prints warning: using the internal BAML toolchain binary directly is not recommended on every invocation — that is expected; ignore/grep it out. The binary name is baml-cli (hyphen), even though the crate is baml_cli.

baml describe — the CLI is the stdlib documentation. Never guess.

The single most important tool for end-to-end work. The stdlib is large (~50 files, crates/baml_builtins2/baml_std/**.baml); rather than guess method names, ask the binary:

$BAML describe baml                 # ← THE FULL PICTURE: every namespace, type & function
                                    #   in the stdlib in one listing (csv, env, errors, fs,
                                    #   http, json, math, net, time, toml, yaml, iter, …)
$BAML describe baml.json            # drill into a namespace → its types + function signatures
$BAML describe Array                # drill into a type → full method list + docs
$BAML describe String --budget 200  # output is line-budgeted; raise --budget to see all methods
$BAML describe <YourSymbol>         # also works on symbols in the loaded project

describe resolves symbols against a project. From inside a project dir it just works; from elsewhere pass --from <project-dir>. Output is capped by --budget (default 30) and tells you "… N more lines (re-run with a higher --budget)" — raise it to see everything. Anything you can't see, describe it; do not guess stdlib names or signatures.

Setting up a project

$BAML init <dir> --name <name>      # scaffolds <dir>/baml.toml + <dir>/baml_src/main.baml
                                    # (refuses to clobber an existing baml.toml)
$BAML new <dir>                     # like init but creates a fresh dir (errors if it exists)

baml.toml minimum is just [package]\nname = "...". Source lives under baml_src/**.baml. An optional [scripts] table aliases baml run invocations (e.g. dev = "-f main").

Running code

# Eval a one-off expression — fastest feedback loop, doubles as a syntax/type check.
# Runs WITHOUT a project; great for probing stdlib behavior in isolation.
$BAML run -e '1 + 2'                                  # → 3
$BAML run -e 'let xs=[3,1,2]; xs.length()'            # → 3
$BAML run -e 'baml.unstable.string(6)'                # → "6"

# Run a named function in the loaded project. The runtime builds a typed clap CLI from the
# function signature and exposes each function as a SUBCOMMAND, so the function name must be
# REPEATED after `--`, then its args as flags:
$BAML run main                                        # simplest for a zero-arg fn (positional target)
$BAML run --function main -- main                     # equivalent explicit form
$BAML run --function greet -- greet --name "Ada"      # scalar args: repeat the fn name, then --flags
$BAML run --function total -- total --json-args '{"xs":[3,1,2]}'   # collection/class/union args: use --json-args
# GOTCHA: bare `$BAML run --function main` (no `-- main`) prints a clap usage screen, not the result.
# GOTCHA: `--json-args` IS a real flag and is REQUIRED for array/map/class/union params (it goes
#         AFTER the repeated function-name subcommand).

# Compile-check the whole project without running:
$BAML check

baml run always prints a Loading … / Checking … / Compiling … preamble; the program's result is printed after. On compile errors it prints rich diagnostics with Error code: E#### and exits non-zero with Cannot run: compilation errors found.

Tests + assertions

$BAML test --list                    # discover tests without running
$BAML test                           # run all; prints PASS/FAIL per test, exits non-zero on failure
$BAML test -i "<testset>::<case>"    # run a single test by id

• A single test needs no wrapper: test "name" { ... }. testset "name" { ... } only groups.
• Assertions live in the assert namespace and throw (panic) on failure: assert.is_true(cond), assert.equal(actual, expected), assert.not_null(v), assert.contains(haystack, needle). A test passes iff its body runs without an uncaught throw.
• A failing assert surfaces as UnhandledThrow { value: Instance { class_name: "baml.panics.UserPanic", … } } with a stack trace pointing into testing/registry.baml.
• LLM functions (client: + prompt:) hit the network — do not rely on live LLM calls in e2e tests. Test the deterministic logic, and for parsing use the generated Fn$parse(raw) companion against a canned string (or baml.json.from_string<T>(...)).

A known-good reference program

function sum_list(xs: int[]) -> string {
  let total = 0;
  for (let x in xs) {        // for-loops need `(let …)` and iterate VALUES
    total += x;
  }
  return "sum=" + baml.unstable.string(total);   // no implicit int→string coercion
}

test "sums inline" {
  assert.equal(sum_list([3, 1, 2]), "sum=6")     // inline call form
}

Language cheat-sheet (the traps that cost the most time)

BAML is expression-oriented and TypeScript-ish with snake_case methods. The five things that bite first — everything else, baml describe it:

1. Class fields are name: type, (trailing comma); construct with Point { x: 1 }. Methods take an explicit self; static factories don't. baml fmt normalizes layout.
2. Last expression in a block is its value (Rust-style). Early exit is return x; (with trailing ;). A no-value function is -> null with a trailing null.
3. for (let x in xs) iterates values and requires let. if / match / blocks are expressions; match (v) { 0 => "a", _ => "b" }.
4. No implicit string coercion — "n=" + 5 will NOT compile; use baml.unstable.string(5). Indexing out of bounds panics — use .at(i) / map .get(k) which return T?. Closures are (x: T) -> R { ... }; the => arrow is match-only. .map/.filter return arrays directly (no .collect()). Map keys must be string.
5. catch arms are type-only and non-exhaustive: f(x) catch (e) { BadInput => fallback }. throws T is part of a function's signature; panics are not catchable.

Recommended workflow

baml describe baml (full picture) → sketch the program → baml run -e / baml check constantly for fast feedback → baml describe <name> whenever you need a signature → baml test → baml fmt baml_src/*.baml before finishing.

Finding & reporting language bugs end-to-end

When something behaves wrong, prove it is the language, not your code:

1. Reduce to the smallest .baml (or baml run -e one-liner) that still shows it.
2. Confirm the intended behavior via baml describe / the stdlib source so you're sure it's a real defect and not a misuse.
3. Record an exact repro: the minimal source, the exact command, observed vs. expected, and the E#### error code or runtime throw. Classify severity: crash (VM/compiler panic or internal error) > wrong-result > spurious-compile-error (rejects valid code) > missing-error (accepts invalid code) > bad-diagnostic.
4. A bug that only reproduces inside one construct (e.g. only inside a test block, only under a generic) is worth noting — the construct boundary is a strong clue to the root cause.

Example of a real defect found this way: inside a test block, a let-bound local does not compare equal to a literal of the same value — test "x" { let r = "x"; assert.equal(r, "x") } fails, while the inline form assert.equal("x", "x") and run -e 'let r="x"; r=="x"' both pass. Tiny source, exact command, observed≠expected, construct-scoped → that's a good report.