Databend Meta Dependency Optimization Benchmark

Date: 2026-05-15

This rerun supersedes the 2026-05-14 result and adds a 1024-client setting.

Scope

This benchmark compares the old Databend ups/main baseline before the Databend Meta dependency update with the current optimized Databend branch after the released Databend Meta tags were merged.

The benchmark target is Databend Meta in a 2-node cluster. The measured workload is upsert_kv.

Binary And Dependency Versions

Current optimized binary:

Baseline binary:

The benchmark client is the current optimized databend-metabench binary for both sides. The current rows were rerun after databend-metabench switched its benchmark-local latency histogram to base2histogram; the baseline rows keep the earlier raw reports. The leader and follower in each run use the same server build.

Dependency Optimization

The optimized branch uses released Databend Meta tags that include the raft-log WAL flush optimizations:

• bounded group-commit wait for WAL flushes;
• configurable flush batch size;
• vectored batch writes;
• WAL flush latency percentiles exposed through raft-log stats;
• richer meta service logging for raft-log and Openraft runtime stats.

Benchmark Setup

All runs use scripts/benchmark/meta-cluster-bench.py with a 2-node Databend Meta cluster. Workload is upsert_kv.

Raw reports and logs were generated under:

/Users/drdrxp/xp/vcs/sessions/stream/bench-reports/databend-meta-final-20260515

Result

Raft-log Stats

The new raft-log stats are available only in the current optimized build. For the 1024-client current rerun, the server-side raft-log and runtime rows use the last stats sample emitted by the server logs; the client-side summary in the Result table is the final metabench summary.

Stage Latency

Interpretation

The current optimized dependency stack improves throughput in the 4, 64, and 256 client runs:

• 4 clients: 172.9 -> 193.8 qps, +12.1%;
• 64 clients: 2807.0 -> 3210.8 qps, +14.4%;
• 256 clients: 8920.3 -> 12127.5 qps, +36.0%.

The 64-client and 256-client runs show the clearest win: the current build removes the slow RPC and slow IO outliers seen in the baseline run, and p99 client latency improves from 45ms -> 30.2ms at 64 clients and from 200ms -> 39.8ms at 256 clients.

The 1024-client current run was rerun as 1024 clients x 156 ops/client, about 6.8s total, after switching databend-metabench from the old manual latency buckets to base2histogram. Client-side qps is slightly lower than the old baseline row (24262.5 -> 23440.1 qps, -3.4%) and p90 client latency is worse (45ms -> 57.7ms). The more accurate client-side p99 is 80.6ms; the previous 300ms value was the old coarse bucket upper bound, not a precise percentile estimate.

On the server side, the rerun still shows a faster persistence stage: submitted->persisted p90 drops from 33916us to 16452us. The remaining 1024-client latency is mainly after persistence: proposed->applied p90 is 56281us, with a large persisted->committed component in the runtime stats.

The persistence path is still a major cost. In the optimized build, submitted->persisted remains roughly 16ms to 20ms at p90 across these runs. The current optimization mainly amortizes the sync cost by batching more writes per WAL flush; it does not eliminate the underlying fsync latency.

Raw Files