Security Policy Benchmark

Local benchmark for measuring Row Access Policy (RAP) and Data Mask Policy performance overhead in Databend.

What It Measures

1. RAP vs WHERE: Overhead of row access policy (auto prewhere pushdown) compared to an equivalent hand-written WHERE clause.
2. Mask vs Function: Overhead of masking policy (auto function replacement) compared to an equivalent hand-written function call in SELECT.
3. Planner Cache: Cold vs warm plan cache latency with and without security policies.

Usage

Against a running Databend instance

# Default: 5M rows, all complexity levels, 5 bench rounds
python3 scripts/security_policy/security_policy_bench.py \
    --dsn "databend://[email protected]:8000/default?sslmode=disable" \
    --output tmp/security_policy_bench_result.json

# Quick run: single complexity, fewer rounds
python3 scripts/security_policy/security_policy_bench.py \
    --complexity simple --bench-rounds 3 --output tmp/quick.json

# Dry run: print generated SQL without executing
python3 scripts/security_policy/security_policy_bench.py --dry-run

With a specific binary (auto start/stop services)

python3 scripts/security_policy/run_security_policy_bench_with_bin.py \
    --query-bin target/release/databend-query \
    --output tmp/result.json

Cross-version A/B comparison

python3 scripts/security_policy/run_security_policy_bench_with_bin.py \
    --query-bin target/release/databend-query \
    --compare-bin /path/to/other/databend-query \
    --output tmp/result.json

This runs the benchmark on both binaries and outputs a diff report. Regressions (>5% slower) are flagged.

CLI Arguments

security_policy_bench.py

run_security_policy_bench_with_bin.py

Metrics

Primary metrics (per query)

• duration_ms: Best-of-N wall time (ms)
• duration_avg_ms: Average wall time across rounds
• all_ms: All individual round timings

Pruning stats (RAP queries)

• segments_before/after: Segment range pruning
• blocks_before/after: Block range pruning
• *_pruned_pct: Percentage pruned

Summary output

• rap_vs_where.overhead_pct: Policy overhead relative to manual WHERE
• mask_vs_func.overhead_pct: Policy overhead relative to manual function

Test Table

CREATE TABLE bench_security (
    id         INT NOT NULL,
    tenant_id  INT NOT NULL,
    email      VARCHAR NOT NULL,
    amount     DECIMAL(18,2) NOT NULL,
    status     VARCHAR NOT NULL,
    created_at TIMESTAMP NOT NULL
) CLUSTER BY (created_at);

• Clustered by time (realistic production pattern)
• 100 distinct tenant_id values scattered across all blocks
• RAP predicates on tenant_id force prewhere filtering (no cluster pruning shortcut)

Policy Complexity Levels