doc/BENCHMARKING.md
This document explains the automated CRUD performance benchmark workflow that runs on every pull request.
The benchmark workflow (crud-bench.yml) automatically runs CRUD benchmarks using crud-bench to measure SurrealDB performance. It posts the results as a PR comment for review.
The benchmark workflow runs automatically on:
The workflow uses a pinned version of crud-bench to ensure consistent benchmarking across all PRs (apples-to-apples comparison). This prevents benchmark variations caused by changes in the benchmarking tool itself.
The specific revision is defined in the CRUD_BENCH_REVISION environment variable in .github/workflows/crud-bench.yml.
When you want to upgrade to a newer version of crud-bench:
Find the target revision:
# Get latest commit from crud-bench main branch
git ls-remote https://github.com/surrealdb/crud-bench.git HEAD
Update the workflow:
.github/workflows/crud-bench.ymlCRUD_BENCH_REVISION in the env sectionTest the upgrade:
Commit and verify:
You can test a different crud-bench version without changing the workflow:
The workflow tests multiple SurrealDB configurations:
These benchmarks test SurrealDB as a server using the binary built from your PR:
surrealdb-memory)surrealdb-rocksdb)surrealdb-surrealkv)These benchmarks test the SurrealDB SDK embedded in the benchmark tool using Cargo patching:
surrealdb + -e memory)surrealdb + -e rocksdb)surrealdb + -e surrealkv)The workflow also benchmarks:
All benchmarks use code from your PR - networked benchmarks use the compiled binary, while embedded benchmarks use the SDK via Cargo patching.
Networked SurrealDB benchmarks require an endpoint field to specify the connection URL. The endpoint field is optional for benchmarks that don't use network connections:
# Networked benchmark - requires endpoint
- name: memory
database: surrealdb-memory
endpoint: ws://localhost:8000
description: SurrealDB with in-memory storage
# HTTP variant (for testing REST API performance)
- name: memory-http
database: surrealdb-memory
endpoint: http://localhost:8000
description: SurrealDB with in-memory storage (HTTP)
# Embedded benchmark - requires endpoint to specify storage
- name: embedded-memory
database: surrealdb
endpoint: memory
description: SurrealDB embedded with in-memory storage
# Local database - no endpoint needed
- name: surrealkv
database: surrealkv
description: SurrealKV
How endpoints are used:
surrealdb-*): Connection URL (e.g., ws://localhost:8000, http://localhost:8000)
ws://127.0.0.1:8000surrealdb): Storage engine specification (e.g., memory, rocksdb:~/data)
ws://127.0.0.1:8000 (not useful for embedded mode, so always specify!)This flexibility allows you to:
The workflow uses a matrix strategy to test each configuration with multiple key types:
Matrix dimensions:
Benchmark parameters per job:
-r flag)Each matrix job runs independently with a clean database state, ensuring accurate performance measurements. Multiple key types ensure performance is measured across different indexing scenarios (small integers vs. long strings).
For each configuration, the benchmark measures:
For each operation, the following metrics are captured:
The workflow measures the following for each benchmark configuration:
Results are posted as a PR comment for manual review and comparison.
After benchmarks complete, the workflow posts a comment on the PR with:
Benchmark results are stored as GitHub Actions artifacts:
These can be downloaded from the workflow run for further analysis or comparison.
Review the results to:
Benchmark results can vary due to:
For more reliable comparisons, consider:
Check the workflow logs for:
If the Python analysis script fails:
The workflow uses parallel matrix execution, so total wall-clock time should be ~5-7 minutes if runners are available.
Build Phase (parallel):
Benchmark Phase (parallel):
If individual jobs are too slow:
-s 10000 in the workflow to -s 5000 or -s 2500-c 12 -t 48 to lower values like -c 8 -t 16If you want to run fewer benchmarks:
exclude: section in the matrixexclude:exclude:
- config: rocksdb
key_type: string250
The workflow uses a three-phase architecture with shared binary builds:
build-surrealdb-binaries (matrix job: 2 parallel builds)
memory: --no-default-features --features "storage-mem,http"rocksdb: --no-default-features --features "storage-rocksdb,http"surreal-binary-{config}build-crud-bench (single job)
.github/tools/.cargo/config.toml) to link crud-bench against local surrealdbcrud-bench-binarybuild-surrealdb-binaries, build-crud-benchneeds_server == true)The workflow ensures all benchmarks test your PR's code through shared build jobs that create binaries once and distribute them to all benchmark jobs:
SurrealDB Binaries (for networked benchmarks)
build-surrealdb-binaries job builds minimal-feature binaries from your PRmemory: In-memory storage + HTTP serverrocksdb: RocksDB storage + HTTP servercrud-bench Binary (for all benchmarks)
build-crud-bench job uses Cargo patching (.github/tools/.cargo/config.toml)surrealdb dependency in crud-bench to use your local SDK codeNetworked Benchmarks (memory, rocksdb):
ws://localhost:8000Embedded Benchmarks (embedded-memory, embedded-rocksdb, surrealkv-local, surrealmx):
The Python script (.github/scripts/analyze_benchmark.py) handles: