SurrealQL Language Tests

This directory contains the surrealql language test suite, consisting of a directory of tests specified in surrealql as well as a command line tool able to run the files and verify their output.

SurrealQL Language Test CLI tool

The surrealql language test CLI tool is the thing which will actually run the test and verify its output. It can be run within this directory with a simple cargo run run (the second run is a command to the tool itself). This will use the test directory as the test root and run all the tests contained within.

The tool can take some flags to change it's behavior. Of note are the filter, and the --results flag.

When you run cargo run run foo you are running the tests with a filter. In this case only test which contain the word foo within its path will be run. This is usefull for running a single or more test you are working on.

The --results flag will change what the tool does with it's result. If you run the test with --results accept the tool will automatically fill in results for any tests which did not specify any results. This is useful for quickly generating expected output for new tests. You can also specify --results overwrite, which will overwrite the specified results of test if they do not match the actual results of the test. This flag should be used sparingly and only after inspecting if the new results are actually valid.

The --backend flag allows you to specify which storage engine to use for running the tests. By default, tests run using the in-memory storage engine (mem or memory). Other supported backends include:

• memory or mem (default): In-memory storage engine, fastest for testing
• rocksdb: RocksDB embedded storage engine (requires backend-rocksdb feature)
• surrealkv: SurrealKV file-based storage engine (requires backend-surrealkv feature)
• tikv: TiKV distributed storage engine (requires backend-tikv feature and a running TiKV cluster)

Note that some backends require the corresponding Cargo feature to be enabled when building:

# Example: Running tests with RocksDB backend
cargo run --features backend-rocksdb run --backend rocksdb

# Example: Running tests with TiKV backend (requires running TiKV cluster at 127.0.0.1:2379)
cargo run --features backend-tikv run --backend tikv

GraphQL tests

Besides .surql files the test directory can also contain .graphql files. These are executed against the dynamic GraphQL schema which surrealdb-core generates from the database catalog — the same code path as the server's /graphql endpoint — instead of the SurrealQL parser. A GraphQL test produces a single result: the response data on success, or the response errors joined into an error string, so [[test.results]] has exactly one entry (value or error).

Because GraphQL requires a configured database, a .graphql test imports a .surql file (see tests/graphql/_schema.surql) which defines DEFINE CONFIG GRAPHQL, the tables, and the data. Imports must always be .surql files. The /** ... */ config comment is not valid GraphQL syntax, so the harness blanks it out (preserving line numbers) before execution.

A .graphql test can additionally set request options in a [graphql] config section:

[graphql]
# GraphQL request variables for the operation.
variables = { personId = "bob", minViews = 10 }
# The operation to execute when the document defines multiple named operations.
operation = "FetchData"

The suite lives under tests/graphql/. GraphQL benches work too — see tests/bench/graphql/ — with the schema generated once up front so the timed iterations measure query execution only.

Benchmarking

The same .surql corpus doubles as a benchmark suite. The entry point is the cargo make bench task: it measures timing by default, or records a flamegraph with --profile. Pass the bench filter (a path substring) and any flags after --:

# measure one bench — times the statement(s) and compares to the saved baseline
cargo make bench -- scans/count

# persist this run as the new comparison baseline
cargo make bench -- scans/count --save

# profile the measured statements with samply (opens an interactive flamegraph)
cargo make bench -- scans/where_integer_in_many_full --profile

# restrict a matrix scan to a single dataset variant (see "dataset matrix" below)
cargo make bench -- scans/where_integer_in_many_full --dataset indexed

# fast, coarse run: shrinks every timing knob ~10x (see "--quick" below)
cargo make bench -- scans/count --quick

# write per-bench results + the comparison verdict to a JSON file
cargo make bench -- scans/count --json results.json

Benches run strictly serially (never in parallel — that would corrupt timings). Each bench warms up, then collects sample_size samples and reports mean / median / std-dev / MAD with confidence intervals, plus a comparison against the saved baseline (improved / regressed / within-noise, with a p-value).

--quick overrides every bench's timing config with much smaller values (250ms warmup, 10 samples, 2s measurement, 10s cap) for a fast, low-power local pass: it reliably surfaces large regressions but is too coarse for few-percent drift.

--json <path> writes a machine-readable report — for each bench its median / mean times and, when a baseline was found, the baseline's median, percentage change, p-value and verdict. Used to render the PR-comparison comment.

--store-url <ws> compares against (and, with --save, writes to) a remote SurrealDB bench datastore instead of the local one. The harness fetch_latestes the most recent stored measurement per bench and compares the current run against it — which is how the PR workflow compares against the nightly main baseline. Requires the bench-remote-store feature.

PR comparison (PR vs nightly main)

The Language bench (PR) GitHub Actions workflow (.github/workflows/language-bench-quick.yml) manages a single sticky comment on every PR:

1. On open, the comment explains how to opt in (add the benchmark label).
2. When the label is added (and on every push while it's set), it benchmarks only the PR head (full run, on the runner-arm-bench pool) and compares it, read-only, against the latest main baseline in the shared bench datastore.

main itself is not benchmarked by this workflow — it's benchmarked by the nightly run (also full, also runner-arm-bench), which --saves main's results to that store. So the baseline is full-quality, measured on the same runner pool, and refreshes whenever nightly re-runs (or nightly is dispatched manually after a notable change). Remove the label to stop.

By default benches run on the in-memory engine. --backend surrealkv (always available) or --backend rocksdb (build with --features backend-rocksdb) run them on the file-backed engines instead, in a temporary directory that's removed afterwards. TiKV is unsupported (it needs an external cluster). Mutating (rebuild = true) benches only run on mem — on a file backend they'd re-import the dataset to disk every iteration — and are skipped (with a notice) otherwise:

cargo make bench -- scans/count --backend surrealkv

[bench] config

[bench]
run = true               # include this file in `bench run`
rebuild = false          # see below; defaults to false
warmup = "2s"            # warmup duration
sample-size = 50         # number of samples (note: kebab-case keys)
measurement-time = "20s" # target total measurement window

rebuild controls datastore lifecycle:

• rebuild = false (default): the datastore is built and imports are run once before timing, then every measured iteration executes the bench statement against that same stable, pre-populated dataset. This is what you want for read-only benches (scans/selects) — it lets the import populate e.g. 100k rows a single time and then measures only the query.
• rebuild = true: the datastore is rebuilt and imports re-run before every iteration. Required for mutating benches (CREATE / UPDATE / DELETE) which need a clean slate each time, otherwise state accumulates across iterations and the workload drifts.

Dataset sharing. Read-only (rebuild = false) benches that resolve to the same dataset — same effective import chain, capabilities, backend, and target namespace/database — are grouped, and their dataset is built and imported once then reused across the whole group (the big records-100000 set, for example, is imported once for the ~26 benches that share it instead of once each). Only the timed execute is measured, so this changes wall-clock only, not the reported numbers. Dataset-matrix variants ([unindexed] vs [indexed]) have different import chains, so they form separate groups with separate datastores.

⚠️ Because grouped benches share one live datastore, a rebuild = false bench whose query mutates state (CREATE/UPDATE/DELETE/…) would corrupt the results of every other bench in its group. Any bench that writes data must set rebuild = true.

The crud-bench ports live under tests/bench/ in the scans/, batches/, and util/ folders. The read-only scan ports set rebuild = false and import a shared dataset from util/ (100k rows mirroring crud-bench/config/bench.toml's [value] shape). They also set [test] run = false so the 100k-row, randomly-generated dataset is exercised only by bench run and not by the regular cargo run run correctness suite (where it would blow the timeout and produce non-deterministic results).

Layout (under tests/bench/):

util/                                # import-only datasets (never run on their own)
  records-100000.surql               # 100k rows, NO indexes (the row generation, in one place)
  records-100000-indexed.surql       # imports records-100000 + adds the standard indexes
  records-100000-fulltext.surql      # imports records-100000 + adds analyzer + BM25 fulltext index
  records-seq-100000.surql           # 100k rows with sequential ids (for record-range scans)
  records-100.surql / records-1000.surql       # small row counts for batch CRUD
  graph-sparse-5000.surql            # 5k person nodes + ~2 `knows` edges each (sparse)
  graph-dense-5000.surql             # 5k person nodes + ~10 `knows` edges each (dense)
  graph-sparse-50000.surql           # 50k person nodes + ~2 `knows` edges each (large, uniform degree)
  graph-hub-50000.surql              # 50k person nodes; 20 fixed hubs w/ 250 edges, rest w/ 2 (degree-skewed)
  embeddings-hnsw-5000.surql         # 5k 8-d vectors + HNSW index
  embeddings-diskann-5000.surql      # 5k 8-d vectors + DiskANN index
  embeddings-flat-5000.surql         # 5k 8-d vectors, no index (brute-force KNN)
  references-5000.surql              # 5k users (+ `friends` link array) + 15k comments whose `author` is a record REFERENCE
  records-100000-perms-full.surql    # imports records-100000 + RECORD access + table PERMISSIONS … FULL (baseline)
  records-100000-perms-where.surql   # imports records-100000 + RECORD access + row-level PERMISSIONS … WHERE
  events-timeseries-100000.surql     # 100k events with timestamps spread across one year (bounded, for bucketing)
  geo-points-100000.surql            # 100k places each with a `location` point spread across the globe
  recursion-functions.surql          # recursive user functions (fn::fib branching, fn::sum_to linear)
  recursion-items-dynamic-1000.surql  # imports recursion-functions + 1000 items with a recursive DYNAMIC (VALUE, write-time) field
  recursion-items-computed-1000.surql # imports recursion-functions + 1000 items with a recursive COMPUTED (read-time) field
scans/
  count / limit / start_limit        # no filter — indexing is irrelevant
  where_*.surql                      # filter scans — run against BOTH datasets (see below)
  aggregate_* / subquery_*           # GROUP BY / GROUP ALL stats / uncorrelated subqueries
  order_by_score_desc / range_scan_* # full sort; contiguous record-id range (± ORDER BY)
fulltext/                            # BM25 search: single / multi-AND / multi-OR (`@@`)
graph/                               # 1-/2-hop (+ counts), filtered, mid-path filter, inbound,
                                     #   bidirectional, recursive, path-collect, shortest-path traversals
                                     #   (each runs against BOTH the sparse and dense graph; two_hop and
                                     #   shortest_path also carry a `large` arm against graph-sparse-50000)
                                     #   plus deep_recursion (6 hops), hub_fanout (250-edge hub node,
                                     #   graph-hub-50000 only), and traverse_aggregate (math::mean over
                                     #   a traversed neighbour set)
vector/                              # KNN: HNSW vs DiskANN vs brute-force
references/                          # record references: reverse (`<~`) traversal/count/filtered, FETCH single + array
mutate/                              # UPDATE/DELETE … WHERE, batch UPSERT, explicit transaction
crud/                                # batch create/read/update/delete (100, 1000)
permissions/                         # per-record table permission cost as a RECORD user: SELECT + UPDATE,
                                     #   each over a row-level `WHERE` vs. `FULL` permission matrix
timeseries/                          # temporal bucketing: GROUP BY time::group(…) by day / hour (± per-bucket stats)
geo/                                 # FULL-SCAN geo functions (no spatial index yet, #219): geo::distance
                                     #   proximity & nearest-N, point-in-polygon containment (INSIDE)
recursion/                           # recursive USER functions (fn::fib, per-row fn::sum_to) + recursive
                                     #   DYNAMIC (VALUE, write-time) vs. COMPUTED (read-time) fields

The derived datasets import the base via [env] imports = ["./records-100000.surql"] and only add their indexes — the 100k-row generation block lives in exactly one file. Imports are resolved transitively, so a scan importing records-100000-indexed runs records-100000 first.

Each conditional (WHERE) scan is a single file that runs against both datasets via the dataset matrix — [bench].datasets is a name → import map:

[bench]
rebuild = false
datasets = { unindexed = "../util/records-100000.surql", indexed = "../util/records-100000-indexed.surql" }

The harness expands this into one run per named dataset (labelled [unindexed] and [indexed]), so the identical query is measured both doing a full table scan and using the index — from one file, no duplication. Run the whole set or one query (both variants run):

cargo make bench -- scans
cargo make bench -- scans/where_integer_eq_full

To run a single matrix variant, add --dataset <name> (matched exactly against the dataset names):

cargo make bench -- scans/where_integer_eq_full --dataset unindexed
cargo make bench -- scans/where_integer_eq_full --dataset indexed

--dataset only affects matrix benches; non-matrix benches (count, limit, fulltext, …) are unaffected.

The permissions/ benches reuse the matrix for a different axis: instead of unindexed-vs-indexed they run a full-vs-where permission pair, and the bench file sets [env].auth to a record user (Session::for_record). Per-record table permissions are only evaluated for record-level actors — owner/editor/viewer sessions take a fast path that skips them entirely (so the existing scans/ and mutate/ benches, which run as owner, never pay this cost). The full variant allows every action with no predicate; the where variant attaches a row-level WHERE clause that is satisfied by every row, so both arms return the same data and the delta isolates the per-record predicate-evaluation cost.

Profiling & flamegraphs

Add --profile to record a flamegraph of the measured statements with samply, built through the profiling cargo profile (release + line-tables-only debug info, unstripped, panic=unwind, thin-LTO for fast iterative rebuilds):

cargo make bench -- scans/where_integer_in_many_full --profile
# saves target/bench-profile/<slug>.json.gz (+ a <slug>.json.syms.json symbol
# sidecar next to it), then:
samply load target/bench-profile/<slug>.json.gz   # interactive flamegraph in the browser

# one variant of a matrix scan (one import, one measured region):
cargo make bench -- scans/where_integer_in_many_full --profile --dataset indexed

The profiler records with --unstable-presymbolicate, which resolves symbols at record time (the binary + its .o debug-map files are present then) into the .syms.json sidecar. samply load reads it automatically, so frames show function names rather than raw hex addresses — keep the sidecar next to the profile. (Attach-mode samply otherwise defers symbolication to load time and can't find the debug info.)

The profile covers only the measured statements — not the dataset setup. For a rebuild = false scan, the one-time 100k-row import (document generation + index build) would otherwise dominate the flamegraph even though it isn't timed. To avoid that, the harness prints a __BENCH_MEASURE_START__ marker on stderr (when BENCH_MARKERS=1) just before the timed loop; the profiler runs the import and warmup un-sampled, waits for that marker, then attaches samply record -p <pid> for the measured region only.

samply is used instead of cargo flamegraph because it samples without sudo on both macOS and Linux (cargo flamegraph relies on dtrace, which is SIP-restricted on macOS). On macOS the script runs samply setup --yes once to codesign the sampler.

Under the hood

cargo make bench runs scripts/bench/bench.sh, which forwards the filter and flags to scripts/bench/measure.sh (timing, the default) or, with --profile, scripts/bench/profile.sh (samply). You can call those scripts directly with the same arguments, or drive the harness binary yourself:

# run every bench (no filter), or one by path substring; flags are all optional
cargo run --features bench -- bench run [--save] [--dataset NAME] [--backend mem] [--quick] [--json PATH] [<filter>]

scripts/bench/optimise.workflow.js is a workflow recipe driving an AI optimisation loop: baseline → inspect the profile → propose & apply one engine change → re-measure → keep only if the harness reports a significant speedup. See the header of that file for how to launch it.

SurrealQL Language Test Format

Language test are plain surrealql files that are parse-able by the normal surrealql parser. Any surrealql file can be a surrealql test, however for a test to be useful it needs to specify a test configuration. This is done by including a special comment, called here after a test comment, in the surrealql file which specifies how the test should be run and their results verified.

Test comments are either single line forward slash comments in the form of //! or a multi line comment in the form of /** */ (note the ! and extra *). When a test is run all the tests comments are concatenated and parsed as TOML.

See below an example of a surrealql test file.

/**
# The env map configures the general environment of the test
[env]
namespace = false
database = false
auth = { level = "owner" }
signin = {}
signup = {}

[test]
# Sets the reason behind this test; what exactly this test is testing.
reason = "Ensure multi line comments are properly parsed as toml."
# Whether to actually run this file, some files might only be used as an import,
# setting this to false disables running that test.
run = true

# set the expected result for this test
# Can also be a plain array i.e. results = ["foo",{ error = true }]
[[test.results]]
# the first result should be foo
value = "'foo'"

[[test.results]]
# the second result should be an error.
# You can error to a string for an error test, then the test will ensure that
# the error has the same text. Otherwise it will just check for an error without
# checking it's value.
error = true
*/

// The actual queries tested in the test.
RETURN "foo";
1 + "1";

SurrealQL Language Test Config Format.

The surrealql test config is specified within the test comment. It is formatted in toml and contains the following sections:

• [test] Defines the information about the test

  • reason a string for detailing why the test exists.
  • run a flag whether this file should be run as a test
  • issue the github issue number to which this test is related.
  • wip whether this test is of a known issue or a work in progress feature
  • version the version requirement for the test running datastore.
  • importing-version the version requirement for the importing datastore.
  • [results] an table which specifies the expected results of the test

• [env] Defines information about how the test should be run.

  • sequential should the test be run without any other test running in parallel
  • clean should the test be run in a completely clean database.
  • namespace the namespace, if any, to run the test in
  • database the database, if any, to run the test in
  • imports a list of files to run before the test.
  • timeout a duration in milliseconds that the test is allowed to take.
  • [login] the login configuration into the dataset.
  • [capabilities] a configuration of database capabilities in which the test should run.

All keys in the config are optional and have default values if not explicitly specified.

[test]

The [test] table specifies the information about the test itself.

[test.run]

Sometimes you want to include a SurrealQL file into the testing suite but not actually care about it's results, for example when the file is intended for as an import for other test files. In this case you can disable running the file as a test by setting [test.run] to false. Defaults to true

[test.wip]

Some tests can be test for a work in progress feature or a known issue or bug that we cannot fix right now. We can still test these these bugs and features by setting [test.wip] to true. Doing so will turn errors in the tests results into warnings which prevents a test run which includes such a test from failing. Furthermore it will also exclude the tests from having it's results automatically be updated by the CI tool when running with --results accept or --results overwrite. Defaults to false

[test.reason] and [test.issue]

Information about the test it's reason to exist and the issue from which it originated. These are mostly just for documentation however when a test is wip and has an issue specified the CLI will then suggest closing the issue if the test succeeded. Both default to None i.e. have no value.

[test.version]

The version requirement for this test. If the test is run with a surrealdb version which does not meet the version requirement it will not be run.

The field expects a string with a version requirement with the semantic versioning format. The same format that is used for specifying rust dependency versions in Cargo.toml.

When importing a file with this field the version is still checked against the importing datastore version and if it does not match the entire test will not run.

Defaults to "*"

[test.importing-version]

The version requirement for the imports of this test. If the import is run with a version which does not meet the version requirement the entire test will not be run.

The field expects a string with a version requirement with the semantic versioning format. The same format that is used for specifying rust dependency versions in Cargo.toml.

This field is not that usefull for normal test as the importing datastore and the test running datastore are always the same. Instead this field is intended for upgrade tests where the upgrading datastore version can differ from the test running datastore.

Defaults to "*"

[test.results]

The test results table specifies the expected out of the test. The command line tool will warn about every test that does not include this table in its configuration. This table can either be a straight table or an array of tables.

Examples:

[test.results]
parsing-error = "foo"

This tests if the test returns a parsing error with the text foo. A test is parsed once and can only return a single parsing error. So when testing for a parsing error only a single result is allowed to be specified.

/**
[test]

[test.results]
parsing-error = """
Invalid function/constant path, did you maybe mean `type::record`
  --> [16:1]
   |
16 | type::thing("person", "one");
   | ^^^^^^^^^^^
"""

*/

type::thing("person", "one");

// Can't add this extra assertion inside the same file
// Must be a separate file with its own [test.results]
// string::slayce();

Note that the following are also allowed:

[test.results]
parsing-error = true

[test.results]
parsing-error = false

Specifying a boolean will check for the presence or absence of a parsing error but will not check the value of the error.

If the test is not intended to return a parsing error it is general best to specify the actual expected output of the tests. A single SurrealQL query can consist of multiple statements which produce either zero or one result. The test allow specifying the expected number and value of the results. Generally specified test results will look like the following:

[[test.results]]
value = "[{ id: foo:bar, name: 'bar' }]"

[[test.results]]
error = "Some error is happening here"

[[test.results]]
error = false

[[test.results]]
error = true

For long string expectations (for example multi-line EXPLAIN output), use a TOML multiline basic string ("""…""") so each logical line is a real newline in the file instead of \n escapes. The value is still SurrealQL parsed like any other value field; literal newlines inside quoted strings are valid SurrealQL. The EXPLAIN text helpers in the database end the string with a newline before the closing quote, so keep that final line break before ' (often with the closing ' alone on the last line) so the expected value matches.

[[test.results]]
value = """
'SelectProject [ctx: Db] [projections: *]
    TableScan [ctx: Db] [table: item, direction: Forward, predicate: score > 15, pre_decode_filter: yes]
'"""

Above we specify that the test should return two results. The first should be a value as described by the value in the string. When specifying a value in TOML strings containing SurrealQL are used. The second result should be an error with the given string as the error text. The third result just specifies that that result should not be an error but gives no details about what the actual value should be. The final result only specifies that the result should be an error but not what the error text should be.

A test with these results will fail if the test returned less or more values and the results are not equal to the given results.

Rough equality

There are some values in SurrealQL which are inherently indeterministic and so would cause problems for test which check if the output is always the same. Generic record-id's is an example, they generally have a ULID key which is always random.

This can be solve with matching expressions but for some common situations it is possible to ignore some parts of the value when matching with the following fields: skip-datetime, skip-record-id-key, or skip-uuid. Setting any of these fields in a [[test.results]] record will skip equality testing the corresponding values.

For example:

[[test.results]]
value = "foo:bar"
skip-record-id-key = true

Will match any record-id as long as the table is foo but the record-id-key is ignored.

Matching expressions

When matching against a value is not possible you can also fall back to running a SurrealQL expression to match the output. Results which should be validated with a matching expression are created by setting the match field on [[test.results]]. The match expression must be a valid surrealql expression which should return a boolean true when the expression found the output to be valid. The matching expression can access the value with either the $result param or the $error param. The latter being defined when the output of the current matched statement was an error, being defined with the text of the error as a string. It is often the case that a matching expression should only match a value or an error but not both. In this case you can set the error field on the same [[test.results]] to either true or false depending on if an error was expected or not. See below for some examples of matching expressions:

# Tests if the statement output was either the string foo or an error: 'An error
# occurred: foo'
[[test.results]]
match = "$result == 'foo' || $error == 'An error occurred: foo'"

# Tests an error with a regex as some parts of the error are non-deterministic.
[[test.results]]
match = "$error = /Found record: `thing:.*` which is not a relation, but expected a  NORMAL/"
# This matching should only match errors.
error = true

# Test whether the field of a result matched the regex
[[test.results]]
match = """
$result.users.test = /DEFINE USER test ON ROOT PASSHASH '\\$argon2id\\$.*' ROLES VIEWER DURATION FOR TOKEN 1h, FOR SESSION NONE/
"""
error = false

[env]

The [env] table specifies the environment in which the test must be run.

[env.clean]

To ensure test run quickly the CLI will generally try to reuse databases between tests. When a test is completed the database and namespace it ran in will be removed so that the next test can be ran in a clean environment. If a test can cause a database to be altered even after the database and namespace where removed then the test should be run within a fully clean database. This can be done with this flag which will cause the test to be run in a freshly create database which will be destroyed after the test completes. Defaults to false.

[env.sequential]

To run test as quickly as possible test the CLI will try to run tests in parallel when possible. If this would cause issues or if this test consume a lot of threads it might be better to ensure that no other test is running at the same time. Setting [env.sequential] will ensure that a test is run when no other test is running. Defaults to false.

[env.namespace] and [env.database]

These keys set the name of the namespace and database the test is run in. These can either be a string or a boolean. When the key is a string the string will be the name for the namespace or database. If the key is false then the test will not be run in a namespace or database. If the key is true it will default to the default namespace and database name: "test". Defaults to true

[env.imports]

An array of string which you can use to specify files to run before running the test. If the path starts with ./ or ../ the path is relative to the directory of the current file otherwise the path is relative to the root of the test directory. Each of these files will be run in a database with full capabilities and the given namespace and database. The test is only run after the files in imports are run to completion.

When importing a test the [test.version] field of the imported test is checked against the version of the importing datastore. If it does not match than the entire test is not run.

This can be used to import a dataset before running queries, importing utility functions, or setting up a database with root access before running permission specific tests.

Defaults to []

[env.backend]

Specifies which backends (storage engines) the test should run on. This is useful when a test is specific to certain backend implementations or when testing backend-specific features like compaction.

The value should be an array of strings with valid backend identifiers:

• "mem": In-memory storage engine
• "rocksdb": RocksDB embedded storage engine
• "surrealkv": SurrealKV file-based storage engine
• "tikv": TiKV distributed storage engine

Behavior:

• If the array is empty (default): The test runs on all backends
• If the array is specified: The test only runs when the selected backend (via --backend CLI flag) is in the list
• If no [env] section exists: The test runs on all backends

Examples:

# Test that only runs on RocksDB
[env]
backend = ["rocksdb"]

# Test that runs on both memory and RocksDB backends
[env]
backend = ["mem", "rocksdb"]

# Test that runs on all backends (empty array)
[env]
backend = []

# Test that runs on all backends (no backend field specified)
[env]
namespace = "test"
# No backend field means run on all backends

This feature is particularly useful for testing backend-specific operations. For example, the ALTER TABLE COMPACT statement behaves differently on different backends: it succeeds on RocksDB but returns an error on the memory backend.

Defaults to [] (runs on all backends)

[env.versioned]

Specifies whether the test requires MVCC versioning to be enabled on the datastore. When set to true, the datastore is created with versioning support, enabling temporal queries using the VERSION clause.

Tests with versioned = true always get a fresh datastore (they cannot reuse the shared datastore pool) since they require different datastore configuration.

Examples:

# Test that requires versioning on memory and SurrealKV backends
[env]
backend = ["mem", "surrealkv"]
versioned = true

# Test with versioning on all backends that support it
[env]
versioned = true

Defaults to false

[env.timeout]

Specifies a duration in milliseconds within which the entire test should finish. This controls the overall test execution time from start to finish. If the test takes longer than the given duration it will be considered an error and it will cause a test run to fail. This key can also be set to false to disable the timeout altogether or true to default to 5 seconds. Defaults to 5000 (5 seconds).

[env.timeout-tikv], [env.timeout-rocksdb], [env.timeout-surrealkv]

Backend-specific timeout overrides in milliseconds. When running tests against a specific backend, these values take precedence over the base timeout. This is useful for tests that are known to be slower on certain backends (e.g., TiKV due to network latency).

Example:

[env]
timeout = 5000           # Timeout for memory backend
timeout-tikv = 10000     # TiKV needs 2x more time due to network latency
timeout-rocksdb = 6000   # RocksDB may need slightly more time for disk I/O

[env.context_timeout]

Specifies a duration in milliseconds for individual query execution within the datastore context. This controls how long each query is allowed to run. If a query takes longer than the given duration, it will be terminated. This key can also be set to false to disable the context timeout altogether or true to default to 5 seconds. Defaults to 5000 (5 seconds).

[env.context-timeout-tikv], [env.context-timeout-rocksdb], [env.context-timeout-surrealkv]

Backend-specific context timeout overrides in milliseconds. Similar to the timeout overrides, these allow setting different query execution limits per backend.

Example:

[env]
context-timeout = 5000           # Default context timeout
context-timeout-tikv = 10000      # TiKV queries may take longer

Note: [env.timeout] and [env.context_timeout] serve different purposes:

• timeout: Controls the entire test execution time (end-to-end)
• context_timeout: Controls individual query execution time within the test

[env.signin] [env.signup]

Specifies how to signin and signup into to the datastore. This field expects a object with fields similar to the fields passed to signin/signup RPC methods. For example:

[env]
signin = """{
	ns: "test",
	user: "ns_user",
	pass: "pass",
 """

Will run a signin to the namespace test with the username ns_user and the password pass.

Signin and signup happen before the test is run but after the imports are run. The imports are always run as root. If an error happens during signin or signup than that error will be the result of a test. This error can be matched against just like a parsing error.

This field is not supported for upgrade tests.

[env.auth]

Specify the authority with which the test is run, unlike [env.signin] and [env.signup] this field bypasses the normal checks and code run to validate signin and signups. This field can be one of 4 different variants:

[env]
auth = { level = "viewer" }

Will authenticate with the viewer role on the datastore root.

[env]
auth = { namespace = "ns", level = "viewer" }

Will authenticate with the viewer role on the namespace ns .

[env]
auth = { namespace = "ns", database = "db", level = "viewer" }

Will authenticate with the viewer role on the namespace ns and the database db.

[env]
auth = { namespace = "ns", database = "db", access = "access_definition", rid = "user:account" }

Will authenticate on the namespace ns and database db as the access method access_definition with the record id user:acount.

This field is not supported for upgrade tests.

[env.capabilities]

This is a table which can be used to specify the capabilities with which the database should be run. This can be used to disable functions, net targets, http routs and scripting just like with the SurrealDB binary/rust SDK. Defaults to all capabilities enabled.

This field is not supported for upgrade tests.

[env.planner-strategy]

Controls which planner strategies the test is executed under. The test runs once per listed strategy, each with its own clean datastore.

Valid strategy values:

• "best-effort-ro": Try the new planner for read-only statements. If the planner returns Unimplemented, silently fall back to the legacy compute executor.
• "all-ro": Require the new planner for all read-only statements. If the new planner cannot handle a non-DDL/DML statement, the test fails with an error instead of silently falling back. DDL/DML statements (CREATE, UPDATE, DELETE, DEFINE, etc.) always fall back regardless of this setting.
• "compute-only": Skip the new planner entirely and always use the legacy compute executor for all statements.

Defaults to ["compute-only", "all-ro"] when omitted, meaning the test runs twice: once under the legacy compute executor and once requiring the new planner.

Examples:

# Run only under the new executor (narrow from the default two-strategy run)
[env]
planner-strategy = ["all-ro"]

# Run only under legacy compute (e.g. features not yet supported by the new planner)
[env]
planner-strategy = ["compute-only"]