Benchmark

This benchmark answers one question for the recommended production shape: at the 1:10 app-to-worker ratio, how does throughput scale as you grow the fleet from 40 to 160 workers? It runs the worker-is-the-sandbox model on a real GKE cluster, against a same-region object store with signed URLs and official piece tarballs served from the CDN.

The shape under test: app tier, Redis job queue, Postgres, S3, and a one-flow-per-worker execution tier.

What's measured

A 4-node synchronous webhook flow that holds the HTTP connection open until the flow returns:

The compute is sub-millisecond by design — everything measured below is orchestration (queueing, callbacks, sandbox boot), which is what actually shapes production latency.

Each fleet size is held at the recommended 1:10 ratio (1 app per 10 workers) and run warm (AP_REUSE_SANDBOX=true) — the engine process is reused between jobs.

Results

Each worker is one sandbox at concurrency 1, hard-capped at 0.5 vCPU / 1 GB. Apps are 1 vCPU / 1 GB. Load concurrency is matched to the worker count so requests don't queue behind the concurrency-1 workers.

What each tier ran — and what it was actually doing

Only the app and worker counts scale (1:10). Postgres and Redis are a single fixed-size pod each — the same for every row below. CPU is the average across three warm load tests; the singletons' figures are the whole pod, app/worker are per pod.

Postgres never crosses ~0.65 of a core and Redis never crosses ~0.17, both far below their caps and flat as the fleet quadruples — they are not absorbing a growing share of anything. Workers sit at ≤0.1 of their 0.5-core cap. No tier approaches saturation, which is exactly why each added worker keeps adding throughput. (The singletons are sized this large on purpose — see Test environment — so they provably stay off the critical path; the default Postgres max_connections=100 would cap the fleet at ~10 apps, which is the artifact behind the earlier "120 cliff".)

How throughput scales

• Warm scales linearly with the fleet. Per-worker throughput stays flat at ~4.5 req/s from 40 to 160 workers, so total throughput tracks the worker count (185 → 410 → 553 → 686; 3.7× for 4× the fleet). The shared Postgres and Redis singletons are not the wall — they sit near-idle at every fleet size (Postgres under 0.6 of a core, Redis under 0.2, both far below their caps), and raising their resources several-fold does not move the curve. The ceiling is the concurrency-1 worker model: each worker is busy for the whole per-flow time — engine run plus the end-of-run run-log persistence it finishes before taking the next job — so fleet throughput is workers ÷ per-flow-time, which is linear in the fleet. (The synchronous response reaches the client sooner than that — it is sent at the response step, before the worker wraps up the log write — so client-perceived latency is lower than the worker-busy time that sets throughput.) Per-flow time carries run-to-run variance (the object-store log-write tail), which is why a single run's curve looks bumpy; the invariant that the per-worker rate holds constant is what shows the scaling is linear. <Note>

Why Production Setup recommends 1:10. Apps at 1 vCPU are cheap relative to the worker fleet, and 1:10 is the warm-headroom margin that keeps the app tier from becoming the wall during bursts. See Production Setup. </Note>

Latency anatomy

Where the worker's milliseconds go — warm at peak (16 app · 160 w):

This is the time the worker is occupied per job — and at concurrency 1 it is what sets throughput (workers ÷ worker-busy-time). The synchronous client sees less: the response is published at the flow's response step, before the worker finishes persisting the run log, so client-perceived latency runs below the worker-busy figure.

Test environment

• Cluster: GKE n2-standard-16 × 10 nodes, europe-west1-b
• Worker: 0.5 vCPU / 1 GB, concurrency 1, SANDBOX_CODE_ONLY (Node fork + isolated-vm)
• App: 1 vCPU / 1 GB
• Object store: same-region GCS bucket (europe-west1) over the S3-interop endpoint, path-style SigV4 presigned URLs (AP_S3_USE_SIGNED_URLS=true)
• Piece bundles: official tarballs served from the Activepieces CDN (AP_USE_CDN_FOR_BUNDLES=true)
• Postgres + Redis: in-cluster singletons, deliberately over-provisioned so they stay off the critical path — Postgres at 3 vCPU / 3 GB with max_connections=2000 (the default 100 would starve the app pools past ~10 apps), durability off, and its data dir on tmpfs; Redis at 2 vCPU / 2 GB with io-threads. Under load both stay near-idle (Postgres <0.6 core, Redis <0.2), confirming the worker tier, not the singletons, is the ceiling.
• Load: hey, concurrency matched to worker count (40/80/120/160) so requests don't queue behind the concurrency-1 workers — latency reflects real service time, not backlog

How to reproduce

benchmark/run-gke.sh [total_requests] [concurrency]

The script mints a worker token, deploys benchmark/k8s-sandbox.yaml to the cluster, runs the load test against the app LoadBalancer, and reports warm throughput and the per-run breakdown from worker-pod logs. Set APP_REPLICAS and WORKER_REPLICAS (keeping the 1:10 ratio) to reproduce any row in the results table.

Benchmark your own installation

Load-test and diagnose your own deployment — no cluster scripts required — with the CLI. It publishes a synchronous flow (webhook trigger → data mapper → return response), fires load at its sync webhook endpoint with autocannon, and returns one self-contained diagnostic bundle you can hand to support.

AP_API_KEY=<key> npx @activepieces/cli@latest benchmark --url https://your-instance.example.com --project-id <id>

Why the numbers are trustworthy. The CLI runs from a different region than your servers, so its client-side latency is polluted by network distance. The numbers that matter are measured server-side instead: the QUEUE/PROVISION/BOOT/RUN split is timed inside the worker, and DB/Redis/S3 round-trips are measured in-region by an admin diagnostics endpoint. Client numbers are shown but marked observational.

Concurrency defaults to your execution slots (Σ AP_WORKER_CONCURRENCY across connected workers) so requests don't queue and you read real service time, not backlog. Comparing two deployments? Match concurrency to each one's own slots — never a fixed number, which makes the smaller one queue.

Reference numbers

A real run against the recommended GKE deployment from the results table: 4 workers @ 0.5 vCPU / 1 GB, concurrency 1, SANDBOX_CODE_ONLY, AP_REUSE_SANDBOX=true, same-region GCS with signed URLs, warm, load = concurrency 4 (= slots) × 200 requests.

Run the CLI against your own deployment and compare tier by tier. A number several times larger localizes the problem: RUN ≫ 200 ms means a heavier flow or a CPU-starved worker; storage ≫ 240 ms means a mis-regioned or throttled object store; a large QUEUE with climbing queue depth means you drove more concurrency than you have slots.

Version & health
  app version : 0.86.2 (latest available: 0.86.2) — all workers match app version
  health      : app-cpu=ok app-ram=ok disk=ok worker-cpu=ok worker-ram=ok db=ok

Infra round-trip (server-measured, in-region — authoritative, not reachable from the CLI)
  database : 21 ms
  redis    : 17 ms
  storage  : 237 ms   (S3/GCS write+read round-trip; same cost is inside every RUN as the end-of-run log backup)
  config   : execution=SANDBOX_CODE_ONLY storage=S3 signedUrls=true sandboxMemKB=1048576 s3=https://storage.googleapis.com/europe-west1
  workers  : 4 connected
    - ONLINE worker-z | 0.5 core | cpu 100.0% | ram 13.8%
    - ONLINE worker-N | 0.5 core | cpu  63.9% | ram 15.2%
    - ONLINE worker-u | 0.5 core | cpu 100.0% | ram 15.5%
    - ONLINE worker-e | 0.5 core | cpu 100.0% | ram 15.5%

Config flags (server-reported)
  EDITION "ce"   DEFAULT_CONCURRENT_JOBS_LIMIT 1000   PROJECT_RATE_LIMITER_ENABLED false
  FLOW_RUN_MEMORY_LIMIT_KB 1048576   WEBHOOK_TIMEOUT_SECONDS 30   FLOW_RUN_TIME_SECONDS 600  ...

Setup
  workers online : 4, execution slots : 4
  [PASS] sandbox mode / reuse sandbox / worker concurrency / worker CPU / worker RAM   (all match the recommended shape)

Network (CLI -> server, cross-region)
  RTT min / p50 : 43.0 / 46.9 ms over 20 probes

conc 4 (= slots)
  throughput   : 10.0 req/s   (200 reqs in 20.0s)
  run outcomes : 200 SUCCEEDED   (server-truth: 200 2xx, 0 non-2xx, 0 errors, 0 timeouts)
  queue depth  : max waiting 3, max active 4, avg waiting 1   (sampled server-side during load)
  worker-measured latency (authoritative — each phase timed inside the worker, 200 runs):
    QUEUE      p50 100 ms   — wait for a free execution slot  (±app↔worker clock skew; cross-check the queue-depth)
    PROVISION  p50   1 ms   — piece install / cache provision
    BOOT       p50   0 ms   — engine fork + Node boot + isolate + socket connect
    RUN        p50/p90 201 / 244 ms   — engine executes the flow, incl. end-of-run S3 log backup
    => queue-wait p50 102 ms vs service p50 201 ms  =>  verdict: service-bound
  observational (CLI-side, cross-region — NOT authoritative): client latency p50/p90/p99 256/311/5638 ms

Storage (log persistence)
  50/50 sampled runs have a persisted log — the worker->storage write path is healthy

Each section answers one question:

Here the read is unambiguous: workers pegged at ~100% of their 0.5-core limit and RUN ≈ 200 ms dominate, while QUEUE (~100 ms at concurrency = slots) and the infra round-trips are small — the deployment is service-bound on worker CPU, so the lever is more/bigger workers, not a code change.

• Exits non-zero if any request fails — usable as a CI gate.
• The benchmark flow stays in your project — delete it when done.