Benchmark

This benchmark answers one question for the recommended production shape: at the 1:10 app-to-worker ratio, how does throughput scale as you grow the fleet from 40 to 160 workers? It runs the worker-is-the-sandbox model on a real GKE cluster, against a same-region object store with signed URLs and official piece tarballs served from the CDN.

Short answer: throughput keeps climbing to 777 req/s at 160 workers, but not proportionally — the per-worker rate peaks at 80 workers and falls about 20% by 160. The workers are not what limits it: they stay under a fifth of their CPU cap throughout, while database CPU grows in step with throughput.

The shape under test: app tier, Redis job queue, Postgres, S3, and a one-flow-per-worker execution tier.

What's measured

A 4-node synchronous webhook flow that holds the HTTP connection open until the flow returns:

The compute is sub-millisecond by design — everything measured below is orchestration (queueing, callbacks, sandbox boot), which is what actually shapes production latency.

Each fleet size is held at the recommended 1:10 ratio (1 app per 10 workers) and run warm (AP_REUSE_SANDBOX=true) — the engine process is reused between jobs.

Results

Each worker is one sandbox at concurrency 1, hard-capped at 0.5 vCPU / 1 GB. Apps are 1 vCPU / 1 GB. Load concurrency is matched to the worker count so requests don't queue behind the concurrency-1 workers. All 160,000 requests across the four tiers returned 200.

What each tier ran — and what it was actually doing

Only the app and worker counts scale (1:10). Postgres and Redis are a single fixed-size pod each — the same for every row below. CPU is sampled every 3 s throughout the measured pass; the singletons' figures are the whole pod, app/worker are per pod.

Read the two right-hand columns differently from the two on the left. Only the worker has a CPU limit (requests == limits == 500m, a hard cap). Postgres, Redis and the app declare CPU requests — a scheduling reservation they may burst above on a node with spare cores — so their numbers show what each tier consumed, not how close it came to a ceiling.

Postgres is by far the fastest-growing tier. It climbs 529m → 2738m as the fleet quadruples — 5.2× the CPU for 4× the workers — and ends up the single largest CPU consumer in the deployment, above the entire 160-worker fleet combined (~14.6 cores across workers vs 2.7 in one database process, but spread over 160 pods versus one). Its cost per unit of work is near-constant (~2.5 millicores per req/s at every tier), so it grows with throughput, not with fleet size, and no configuration makes it stop growing. Redis behaves the same way (123m → 781m).

By contrast the workers — the only hard-capped tier — sit at ≤0.1 of their 0.5-core cap at every fleet size, and apps hold steady at ~0.52–0.61 of a core per pod because the 1:10 ratio adds app capacity in step with the load.

How throughput scales

Adding workers always adds throughput, but the return per worker falls after ~80. Total throughput goes 213 → 484 → 641 → 777 (3.6× for 4× the fleet). Per worker that is 5.3 → 6.1 → 5.3 → 4.9: the rate peaks at 80 workers and then gives back about 20%.

Two things are happening at once:

• Below ~80 workers the fleet is still filling out. 40 workers cannot push enough concurrent work to use the tiers behind them, so the per-worker rate is not yet at its best. This is why the 40→80 step is super-linear (2.27× throughput for 2× the fleet).
• Past ~80 workers, per-flow time stretches. The measured sandbox run phase goes 146 ms at 80 workers to 181 ms at 160, so each worker completes fewer flows per second even though it is nowhere near its CPU cap. Something shared is absorbing the extra concurrency, and database CPU is the tier visibly growing with it (see the caveat above — growth is measured, causation is not).

The concurrency-1 worker model sets the shape: each worker is busy for the whole per-flow time — engine run plus the end-of-run run-log persistence it finishes before taking the next job — so fleet throughput is workers ÷ per-flow-time. That is linear in the fleet only while per-flow time holds constant, and past ~80 workers it does not. (The synchronous response reaches the client sooner than that — it is sent at the response step, before the worker wraps up the log write — so client-perceived latency is lower than the worker-busy time that sets throughput.)

What this means for sizing: worker count is the right lever up to roughly this scale, but do not read the worker fleet as the only dial — the shared tiers behind it grow with throughput and need to grow with you.

Latency anatomy

Where the worker's milliseconds go — warm at peak (16 app · 160 w), averaged over all 64,000 measured runs:

Warm provisioning and boot are effectively free — the piece cache is on local disk and the engine process is reused, so essentially the entire worker-busy time is the flow run itself. That figure grows with the fleet (167 ms at 40 workers, 146 ms at 80, 167 ms at 120, 181 ms at 160), which is the per-flow stretch behind the falling per-worker rate above.

This is the time the worker is occupied per job — and at concurrency 1 it is what sets throughput (workers ÷ worker-busy-time). The synchronous client sees less: the response is published at the flow's response step, before the worker finishes persisting the run log, so client-perceived latency runs below the worker-busy figure. At peak the in-cluster client measured p50 180 ms, p95 288 ms, p99 652 ms.

Test environment

• Cluster: GKE n2-standard-16 × 10 nodes, europe-west1-b
• Worker: 0.5 vCPU / 1 GB, concurrency 1, SANDBOX_CODE_ONLY (Node fork + isolated-vm). Idle RSS ~145 Mi
• App: 1 vCPU / 1 GB
• Object store: same-region GCS bucket (europe-west1) over the S3-interop endpoint, path-style SigV4 presigned URLs (AP_S3_USE_SIGNED_URLS=true)
• Piece bundles: official tarballs served from the Activepieces CDN (AP_USE_CDN_FOR_BUNDLES=true)
• Postgres + Redis: in-cluster singletons, deliberately generous — Postgres requesting 3 vCPU / 3 GB with max_connections=2000 (the default 100 would starve the app pools past ~10 apps), durability off, and its data dir on tmpfs; Redis requesting 2 vCPU / 2 GB with io-threads. Neither has a CPU limit, so both may burst above their request; Postgres consumed 2738m at the top tier
• Load: hey run inside the cluster, against the app Service, concurrency matched to worker count (40/80/120/160) so requests don't queue behind the concurrency-1 workers — latency reflects real service time, not backlog. 400 requests per worker per tier (16k/32k/48k/64k), preceded by an unmeasured warmup pass

How to reproduce

benchmark/run-gke.sh [total_requests] [concurrency]

The script mints a worker token, deploys benchmark/k8s-sandbox.yaml to the cluster, runs the load test from a pod inside the cluster, and reports warm throughput and the per-run breakdown from worker-pod logs. Set APP_REPLICAS and WORKER_REPLICAS (keeping the 1:10 ratio) to reproduce any row in the results table.

Benchmark your own installation

Load-test and diagnose your own deployment — no cluster scripts required — with the CLI. It publishes a synchronous flow (webhook trigger → data mapper → return response), fires load at its sync webhook endpoint with autocannon, and returns one self-contained diagnostic bundle you can hand to support.

AP_API_KEY=<key> npx @activepieces/cli@latest benchmark --url https://your-instance.example.com

The CLI provisions a throwaway project for the run (with a high concurrency cap so a project rate limiter can't queue-throttle the numbers) and deletes it when finished — nothing is left behind, and it never touches your real projects. The API key must be a platform-admin key.

Why the numbers are trustworthy. The CLI runs from a different region than your servers, so its client-side latency is polluted by network distance. The numbers that matter are measured server-side instead: the QUEUE/PROVISION/BOOT/RUN split is timed inside the worker, and DB/Redis/S3 round-trips are measured in-region by an admin diagnostics endpoint. Client numbers are shown but marked observational.

Concurrency defaults to your execution slots (Σ AP_WORKER_CONCURRENCY across connected workers) so requests don't queue and you read real service time, not backlog. Comparing two deployments? Match concurrency to each one's own slots — never a fixed number, which makes the smaller one queue.

Reference numbers

A real run against a small deployment of the recommended shape — 4 workers @ 0.5 vCPU / 1 GB, concurrency 1, SANDBOX_CODE_ONLY, AP_REUSE_SANDBOX=true, same-region GCS with signed URLs, warm, load = concurrency 4 (= slots) × 200 requests. This is a separate, deliberately tiny deployment used to show what the CLI's output looks like; it is not a row in the results table above, and its cross-region client latency is not comparable to those numbers.

Run the CLI against your own deployment and compare tier by tier. A number several times larger localizes the problem: RUN ≫ 200 ms means a heavier flow or a CPU-starved worker; storage ≫ 240 ms means a mis-regioned or throttled object store; a large QUEUE with climbing queue depth means you drove more concurrency than you have slots.

Version & health
  app version : 0.86.2 (latest available: 0.86.2) — all workers match app version
  health      : app-cpu=ok app-ram=ok disk=ok worker-cpu=ok worker-ram=ok db=ok

Infra round-trip (server-measured, in-region — authoritative, not reachable from the CLI)
  database : 21 ms
  redis    : 17 ms
  storage  : 237 ms   (S3/GCS write+read round-trip; same cost is inside every RUN as the end-of-run log backup)
  config   : execution=SANDBOX_CODE_ONLY storage=S3 signedUrls=true sandboxMemKB=1048576 s3=https://storage.googleapis.com/europe-west1
  workers  : 4 connected
    - ONLINE worker-z | 0.5 core | cpu 100.0% | ram 13.8%
    - ONLINE worker-N | 0.5 core | cpu  63.9% | ram 15.2%
    - ONLINE worker-u | 0.5 core | cpu 100.0% | ram 15.5%
    - ONLINE worker-e | 0.5 core | cpu 100.0% | ram 15.5%

Config flags (server-reported)
  EDITION "ce"   DEFAULT_CONCURRENT_JOBS_LIMIT 1000   PROJECT_RATE_LIMITER_ENABLED false
  FLOW_RUN_MEMORY_LIMIT_KB 1048576   WEBHOOK_TIMEOUT_SECONDS 30   FLOW_RUN_TIME_SECONDS 600  ...

Setup
  workers online : 4, execution slots : 4
  [PASS] sandbox mode / reuse sandbox / worker concurrency / worker CPU / worker RAM   (all match the recommended shape)

Network (CLI -> server, cross-region)
  RTT min / p50 : 43.0 / 46.9 ms over 20 probes

conc 4 (= slots)
  throughput   : 10.0 req/s   (200 reqs in 20.0s)
  run outcomes : 200 SUCCEEDED   (server-truth: 200 2xx, 0 non-2xx, 0 errors, 0 timeouts)
  queue depth  : max waiting 3, max active 4, avg waiting 1   (sampled server-side during load)
  worker-measured latency (authoritative — each phase timed inside the worker, 200 runs):
    QUEUE      p50 100 ms   — wait for a free execution slot  (±app↔worker clock skew; cross-check the queue-depth)
    PROVISION  p50   1 ms   — piece install / cache provision
    BOOT       p50   0 ms   — engine fork + Node boot + isolate + socket connect
    RUN        p50/p90 201 / 244 ms   — engine executes the flow, incl. end-of-run S3 log backup
    => queue-wait p50 102 ms vs service p50 201 ms  =>  verdict: service-bound
  observational (CLI-side, cross-region — NOT authoritative): client latency p50/p90/p99 256/311/5638 ms

Storage (log persistence)
  50/50 sampled runs have a persisted log — the worker->storage write path is healthy

Each section answers one question:

Here the read is unambiguous: workers pegged at ~100% of their 0.5-core limit and RUN ≈ 200 ms dominate, while QUEUE (~100 ms at concurrency = slots) and the infra round-trips are small — the deployment is service-bound on worker CPU, so the lever is more/bigger workers, not a code change.

• Exits non-zero if any request fails — usable as a CI gate.
• The throwaway project (and its benchmark flow) is deleted automatically when the run finishes.