DEVELOP

Overview

This app runs repeatable load tests against multiple connectors (Bun+Postgres, CockroachDB, SQLite, Supabase, Convex, SpacetimeDB, etc.).

Each run:

• Loads a scenario from src/tests/<test-name>/
• Runs it against one or more connectors
• Writes a JSON report into ./runs/ with TPS and latency stats

Demo Mode

Run a quick performance comparison:

pnpm run demo

The script will:

• Check that required services are running (prompts you to start them if not)
• Seed databases with test data
• Run benchmarks at high contention
• Display animated results comparing your chosen systems (SpacetimeDB and Convex by default, as both are simple to run locally)

Options:

• --seconds N - Benchmark duration (default: 300)
• --concurrency N - Concurrent connections (default: 64)
• --alpha N - Contention level (default: 1.5)
• --systems a,b,c - Systems to compare (default: convex,spacetimedb)
• --stdb-compression none|gzip - SpacetimeDB client compression mode (default: none)
• --skip-prep - Skip database seeding
• --no-animation - Disable animated output

demo always runs the built-in test-1 scenario. Use bench if you need to choose a test explicitly. demo uses --systems; bench uses --connectors.

Prerequisites

• Node.js ≥ 22.x
• pnpm installed globally
• Rust (required for SpacetimeDB benchmarks) -- install
• Docker for local Postgres / Cockroach / Supabase
• Local/Cloud Convex

From a fresh clone:

pnpm install

Configuration (.env)

Copy .env.example to .env and adjust.

Seeding / verification:

• SEED_ACCOUNTS – number of accounts to seed (if unset, code defaults to 100_000)
• SEED_INITIAL_BALANCE – starting balance per account
• VERIFY – enable extra verification passes when non-zero
• ENABLE_RPC_SERVERS – flag used by scripts that start the RPC benchmark servers

Runtime toggles:

• USE_DOCKER – 1 = run Docker Compose for Postgres / CockroachDB; 0 = skip
• SKIP_PG – 1 = don't init Postgres in prep
• SKIP_CRDB – 1 = don't init CockroachDB in prep
• SKIP_SQLITE – 1 = don't init SQLite in prep
• SKIP_SUPABASE – 1 = don't init Supabase in prep
• SKIP_CONVEX – 1 = don't init Convex in prep

Throughput is counted from successful operations that the benchmark client observes completing inside the configured test window for every connector, including SpacetimeDB.

PostgreSQL / CockroachDB:

• PG_URL – Postgres connection string
• CRDB_URL – CockroachDB connection string

SQLite:

• SQLITE_FILE – path to the SQLite file
• SQLITE_MODE – tuning preset for the SQLite connector

SpacetimeDB:

• STDB_URL – WebSocket URL for SpacetimeDB
• STDB_MODULE – module name to load (e.g. test-1)
• STDB_MODULE_PATH – filesystem path to the module source (for local dev)
• STDB_COMPRESSION – SpacetimeDB benchmark client compression (none or gzip)
• STDB_CONFIRMED_READS – 1 = force confirmed reads on, 0 = force them off

Supabase:

• SUPABASE_URL – Supabase project URL
• SUPABASE_ANON_KEY – Supabase anon/public key
• SUPABASE_DB_URL – Postgres connection string for the Supabase database

Convex:

• CONVEX_URL – Convex deployment URL
• CONVEX_SITE_URL – Convex site URL
• CLEAR_CONVEX_ON_PREP – Convex prep flag (clears data when enabled)
• CONVEX_USE_SHARDED_COUNTER – flag for using the sharded-counter implementation

PlanetScale:

• PLANETSCALE_PG_URL – Postgres connection string
• PLANETSCALE_RPC_URL – PlanetScale RPC server URL (default: http://127.0.0.1:4104)
• SKIP_PLANETSCALE_PG – 1 = skip PlanetScale in prep

Bun / RPC helpers:

• BUN_URL – Bun HTTP benchmark server URL
• BUN_PG_URL – Postgres connection string for the Bun benchmark service

RPC benchmark servers:

• PG_RPC_URL – HTTP URL for the Postgres RPC server
• CRDB_RPC_URL – HTTP URL for the CockroachDB RPC server
• SQLITE_RPC_URL – HTTP URL for the SQLite RPC server

PlanetScale configuration

Create a Postgres database on PlanetScale and set PLANETSCALE_PG_URL (and PLANETSCALE_RPC_URL if the RPC server runs elsewhere) in .env. Reported results used PS-2560 (32 vCPUs, 256 GB RAM).

Setup

Generate bindings (first time after clone)

SpacetimeDB module bindings:

cd spacetimedb
cargo run -p spacetimedb-cli -- generate --lang typescript --out-dir ../module_bindings --module-path . -y
cd ..

(Or use spacetime generate ... if the CLI is installed.)

Convex generated files:

cd convex-app
npx convex dev
# Wait for it to generate files, then Ctrl+C
cd ..

Start services

1. Start SpacetimeDB (cargo run -p spacetimedb-cli -- start or spacetime start)
2. Start Convex (inside convex-app run npx convex dev)
3. Init Supabase (run supabase init) inside project root.
4. pnpm run prep to seed the databases.
5. pnpm run bench to run the test against all connectors.

Commands & Examples

Run a test

pnpm run bench [test-name] [--seconds N] [--concurrency N] [--alpha A] [--connectors list] [--stdb-compression none|gzip]

Examples:

# Default test (test-1), default args
pnpm run bench

# Explicit test name
pnpm run bench test-1

# Short run, 100 concurrent workers
pnpm run bench test-1 --seconds 10 --concurrency 100

# Heavier skew on hot accounts
pnpm run bench test-1 --alpha 2.0

# Enable gzip for the SpacetimeDB benchmark client
pnpm run bench test-1 --connectors spacetimedb --stdb-compression gzip

# Only run selected connectors
pnpm run bench test-1 --connectors spacetimedb,sqlite_rpc

# Sweep alpha values for a connector set
pnpm run bench test-1 --alpha 0,1.5 --connectors postgres_rpc,bun --seconds 300

# Sweep contention (alpha) for a single connector: start,end,step,concurrency
pnpm run bench test-1 --connectors cockroach_rpc --contention-tests 0,1.5,0.5,64

# Sweep concurrency for a single connector: start,end,factor,alpha
pnpm run bench test-1 --connectors cockroach_rpc --concurrency-tests 16,512,2,1.5

CLI Arguments

• test-name (positional)

  • Name of the test folder under src/tests/
  • Default: test-1

• --seconds N

  • Duration of the benchmark in seconds
  • Default: 300

• --concurrency N

  • Number of workers / in-flight operations
  • Default: 64

• --alpha A

  • Zipf alpha parameter for account selection (hot vs cold distribution)
  • Default: 1.5

• --connectors list

  • Optional, comma-separated list of connector system names

  • Example:

    bashCopy

    --connectors spacetimedb,sqlite_rpc,postgres_rpc
    

  • If omitted, all connectors for that test are run

  • The valid names come from tc.system in the test modules and the keys in CONNECTORS

  • Valid names: convex, spacetimedb, bun, postgres_rpc, cockroach_rpc, sqlite_rpc, supabase_rpc, planetscale_pg_rpc

• --systems list

  • Alias for --connectors in bench mode

• --runs N

  • Repeat each (connector, alpha) combination N times
  • Default: 1

• --prep-between-alphas

  • Run pnpm run prep before each (connector, alpha) combination

• --contention-tests start,end,step,concurrency

  • Sweep Zipf alpha values for one connector

• --concurrency-tests start,end,factor,alpha

  • Sweep concurrency values for one connector

• --bench-pipelined / --no-bench-pipelined

  • Force pipelining on or off across connectors

• --max-inflight-per-worker N

  • Max in-flight requests per worker when pipelining is enabled
  • Required when --bench-pipelined is enabled

• --log-errors

  • Log per-operation errors during runs

• --verify-transactions

  • Run connector verification at end of run

Running in Docker

You can also run the benchmark via Docker instead of Node directly:

docker compose run --rm bench -- --seconds 5 --concurrency 64 --alpha 1 --connectors convex

If using Docker, make sure to set USE_DOCKER=1 in .env, verify docker-compose env variables, verify you've run supabase init, and run pnpm run prep before running bench.

Output

Every run writes a JSON file into ./runs/:

• Directory: ./runs/
• Filename: <test-name>-<connector>-a<alpha>-<timestamp>.json
  • Example: test-1-postgres_rpc-a1.5-2025-11-17T16-45-12-345Z.json

For rollup tables, compute steady-state stats after a 30-second warmup window (tSec >= 30). The scripts/bench-stats.py default matches this (--warmup-sec 30).

Point your visualizations / CSV exports at ./runs/ and you're good.