brain/knowledge/execution-runtime/workers.md
Separate Node processes that poll the app for jobs and execute flows/triggers. The worker is the sandbox: each forks the engine in-process via @activepieces/sandbox (createSandboxRuntime) — no separate sandbox pool. Destination model is one box per worker (concurrency 1), scaling out horizontally with small-capped replicas so an OOM kills one job, not a shared pool. A transitional mode still honors AP_WORKER_CONCURRENCY.
The deep Resolver/Runtime concurrency and bundle-caching model lives on the Execution Runtime page — see it for provision/acquire/release, flow-bundle caching, and code-step build stubs.
FETCH_WORKER_SETTINGS, get a WorkerSettingsResponse (incl. APP_VERSION), and the app registers an RPC server (WorkerToApiContract) per socket.poll(), not pushed. Handing out a job moves it to BullMQ active, so the app records it under the polling socket.id (jobAssignmentTracker, keyed queue+jobId). Worker executes → periodic extendLock → completeJob clears the assignment.connectionGeneration++ stops the loops; the app returns that socket's still-held jobs to the queue (releaseConnectionJobs(socket.id)) so they don't sit orphaned in active (the "Job stalled" storm); the worker aborts its in-flight runtime. Reconnect recreates the runtime fresh (kills any lingering job rather than colliding on the reused box).stop() (SIGTERM) drains in-flight jobs first (≤25s); the stalled-scan is the backstop for abrupt death.AP_CONTAINER_TYPE (APP / WORKER / WORKER_AND_APP) picks what docker-entrypoint.sh starts. AP_REUSE_SANDBOX reuses the engine process between jobs.AP_WORKER_GROUP_ID + AP_PROJECT_WORKER) — one routing primitive; the flag, not a prefix, encodes scope. AP_PROJECT_WORKER=true (default) → project scope, polls project-<label>-jobs, routes only EXECUTE_FLOW/EXECUTE_WEBHOOK. false → platform scope, polls platform-<id>-jobs, routes all job types. Empty id = shared queue. Per-project routing is EE, gated behind platform_plan.workerGroupsEnabled.concurrency === 1 primes the sandbox to the full container RAM (cgroup-aware) via primeFullContainerMemory().docker-entrypoint.sh launches the bootstrap scripts with plain node --enable-source-maps (removed the pm2-runtime + /tmp/ecosystem.config.js machinery). APP/WORKER exec a single node as PID 1; WORKER_AND_APP runs both and, if either exits, kills the other and exits non-zero so the orchestrator restarts the whole container. This drops PM2's in-container crash/OOM restart: an OOM-kill no longer silently recycles a child every ~4 min behind a RestartCount: 0 (the 2026-07-26 wedge's supply of retry attempts — see below); the container now dies and is rescheduled instead. The historical incident notes below still describe the old PM2 behavior as it happened.versionsAreCompatible (fail-closed — undefined or UNKNOWN_VERSION '0.0.0' is treated incompatible). App withholds jobs from a mismatched worker (poll returns null); worker pauses polling 10s. Ordinary mismatch self-heals on convergence; a read failure does not (cached for process life) and pages on-call once at startup via assertReleaseReadable.process.cwd()/package.json (deploy-root), not a workspace file. Two failed reads are treated incompatible on purpose (not "same release").JobPayload (inline or ref fileId), forwarded unchanged; there is no worker→API payload-fetch RPC.GET /v1/health/system returns a release block (skew across connected workers); /v1/worker-machines/queue-metrics (all editions) feeds KEDA autoscaling.distributedLock, or a system job — it gets a plain unref'd timer and must be idempotent. AP_CONTAINER_TYPE=WORKER boots only packages/server/worker: no Fastify app, no TypeORM connection, no Redis client, and no Redis env (confirmed in production — workers reach the queue only over Socket.IO). distributedLock lives in packages/server/api/src/app/database/redis-connections.ts, and worker's package.json depends on sandbox/server-utils/shared/core-* but not api, and carries neither ioredis nor bullmq — reaching the lock inverts the dependency graph and pulls Fastify, TypeORM and BullMQ into the worker bundle. It is also a trust-boundary change: a worker's only credential is a scoped AP_WORKER_TOKEN, which is what lets it run on a machine not trusted with the platform's queue. And a system job is the wrong shape regardless of Redis — systemJobsSchedule(...).startWorker() runs in the app process, once cluster-wide, so it cannot touch a worker's local disk (Helm's default workloadType: rollout shares one RWO PVC, but statefulset gives each pod its own and the app need not mount it at all). Anything sweeping local state therefore runs on every replica simultaneously by design: make each step idempotent and recompute targets from the live filesystem rather than accumulating state, instead of reaching for a lock. actionRunCache.sweep is the worked example — see [[action-run]] and decision 000016.SANDBOX_CODE_ONLY + concurrency 1 = one immortal engine child allowed to grow to the whole box. These are two safe-looking settings that only misbehave together. canReuseSandbox() (sandbox-manager.ts) returns true for SANDBOX_CODE_ONLY and UNSANDBOXED only, so the engine child is reused forever and release() is a no-op — the process never resets between jobs. At concurrency === 1, primeFullContainerMemory() then overrides the operator's AP_SANDBOX_MEMORY_LIMIT with total container RAM, so that immortal child forks with --max-old-space-size = the entire box. V8 feels no pressure and never returns memory. Measured on 0.86.3, arm64, 16 GB box, AP_SANDBOX_MEMORY_LIMIT=512: the log reads fullContainerMemoryKb: 16332416 (the 512 is silently discarded) and the single sandbox RSS climbs 172 → 235 → 241 → 373 → 375 MB across 6.2k runs — monotonic, never released. In the process-based modes the sandbox is invalidated after every job, so the same workload shows no growth; that is why this only ever gets reported as a SANDBOX_CODE_ONLY "leak". At the default AP_WORKER_CONCURRENCY=5 priming does not fire, and you instead get five persistent children honoring the 512 MB cap (~1.9 GB steady, plateauing) — higher floor, but bounded. Note AP_FLOW_WORKER_CONCURRENCY does not control this; AP_WORKER_CONCURRENCY does, and it defaults to '5' (worker/src/lib/config/configs.ts).cacheState deliberately holds nothing in memory — do not add a memo back. Disk is the only cache (cache-state.ts); every getOrSetCache reads cache.json. That looks wasteful and is not: measured on a cloud worker, a typical bundle's cache.json is 39 KB and costs 0.26 ms to read, against flow runs measured in seconds. The one case where it is not free is a multi-MB bundle (47 MB → ~220 ms), but flowBundleStore.tryFetch already JSON.parses that string twice per run (~123 ms each) via the cacheMiss predicate and again on the result, so a memo never saved the dominant cost anyway. It was an unbounded Record<string, CacheMap> keyed by folder path until Aug 2026, and the failure that caused is the shape to watch for. Unbounded, it was harmless for pieces-metadata (5 folders) and fatal for flow-bundle-store.ts and flow-cache.ts, which put flowVersionId in the path — cardinality became one entry per flow version the worker ever touched, and each bundle value is the entire serialized manifest (flowVersion + pieces + all compiled code), kept as a raw string that is re-parsed on every read anyway. Measured on cloud 0.87.0 before the fix: a worker retained 545 manifests = 170 MB of a ~330 MB heap, single strings up to 90 MB; the host's shared cache held 19,293 bundle dirs / 1.6 GB (~3 GB as UTF-16), and a shared worker walks all of it because it pulls jobs from every project. Symptom is "memory correlates with pod age". Off-heap stays flat ~100 MB — if RSS grows and Used Heap Size grows with it, this is the leak, not native/isolated-vm. Retainer path to look for in a snapshot: Object → property:/…/cache/v13/bundles/<flowVersionId> → property:<flowVersionId> → string. Sizing rule for "will tenant X hit this": count bytes, not flows. The corpus is wildly skewed — median bundle 6 KB, mean 77 KB, p99 1.2 MB, max 47 MB; 95.5% of bundles are under 64 KB and hold 10.5% of the bytes, while the 59 bundles over 4 MB (0.3%) hold 55.8%. So 1,000 median flows is 11 MB and harmless, while ten code-heavy flows is ~400 MB. Model any tenant as heap ≈ 120 MB baseline + Σ(manifest bytes for every distinct flow version executed) × 1.0–1.9 (the ×1.9 is V8's two-byte string case, measured: 47.1 MB on disk → 90.3 MB in heap). Note the unit is the flow version, so republish churn multiplies it.AP_REUSE_SANDBOX=true gives you the immortal engine on SANDBOX_PROCESS too, and the memory it holds is require.cache — bounded, not a leak. The bullet above reads as if the process-sandboxed modes are safe because canReuseSandbox() only returns true for SANDBOX_CODE_ONLY/UNSANDBOXED; they are not. That function checks if (!isNil(settings.REUSE_SANDBOX)) return settings.REUSE_SANDBOX === 'true' first, so the env var wins over the mode. Measured Aug 2026 on a 0.88.1 dedicated cloud host (4 × 1 GiB / 0.5 cpu, SANDBOX_PROCESS, concurrency 1, AP_SANDBOX_MEMORY_LIMIT unset, worker at --max-old-space-size=768): memory is a step function of the distinct module set loaded and it converges. Two engines sampled with a forced GC before each reading — a loaded one held 2,615 modules / 9 resident shared copies / 517 MB post-GC heap / 668 MB RSS flat across 351 s, and a fresh one held 8 modules / 46 MB / 183 MB RSS flat across 842 s. Idle time adds nothing. So "memory correlates with pod age" here is really "correlates with how many distinct piece packages this engine has served". The retainer is Module._cache: full paths to the GC root run zod schema ← property:ShortTextProperty ← property:exports in "Module" ← property:<abs module path> in the 2,531-entry cache object ← property:_cache, there is no Zod global registry and no per-run accumulation, and the cost is ~3,400–4,000 Zod schema objects per resident copy of @activepieces/shared (plus ~285 per pieces-framework copy). Two method notes that cost hours if you get them wrong: attribute by retainer, not node name (name-based attribution finds 4.2 MB of a 642 MB heap and points nowhere), and use post-GC heapUsed + module count as the bounded/unbounded discriminator — RSS alone flattens against V8's own --max-old-space-size ceiling and proves nothing.shared copies → 668 MB, 2,532 modules / 7 copies → 807 MB — and the heavy end plus a ~150 MB worker plus overhead exceeds a 1 GiB cgroup, so the kernel kills at 827 / 842 / 853 MB anon-rss while V8 (capped at 1024 MB, larger than the whole container) never feels pressure. Whether a given container dies is luck-of-the-draw on its flows, which is why sibling containers on one host read oom_kill 6 / 5 / 3 / 0. Priced out: @activepieces/shared is 208.2 MB of a 642.6 MB heap across 7 copies (~33 MB each) plus 8.0 MB for 4 pieces-framework copies — together 97% of everything attributable to require.cache (222.5 MB; the remaining 65% reaches the root via shorter V8-internal paths and so under-attributes modules, making 208 MB a floor). It is mostly not the custom pieces: /root/common (marketplace) holds 5 copies = 132.5 MB against /root/custom_pieces 2 copies = 75.7 MB, because every @europe-express/* piece pins the same shared while old marketplace piece versions each pin a different one (webhook→0.92.0/0.76.7, hubspot→0.87.1/0.37.0, crypto→0.86.0, subflows→0.74.0). Collapsing to one resident copy frees ~170 MB and moves the heavy plateaus back under the cgroup — the highest-leverage fix, ahead of turning reuse off. Related: AP_USE_CDN_FOR_BUNDLES was unset on this host and only 7 of 45 installed piece folders carried a src/bundle.cjs, which is why the copies exist at all; bundles are not free either, the three largest heap objects were 4.6 MB external source strings held twice each.NODE_OPTIONS=--max-old-space-size and the engine child's --max-old-space-size (from SANDBOX_MEMORY_LIMIT, default 1048576 KB = 1024 MB in api/.../system.ts) are independent ceilings in the same container, plus ~90 MB for pm2 and isolate. Cloud prod ran 768 + 1024 + 90 ≈ 1.9 GB of permitted heap in a 1 GiB container: V8 never felt pressure below 768 MB, so the worker walked past the cgroup and the kernel killed it at ~960 MB anon-rss. Because pm2 restarts it, the container still reads "Up" and docker stats looks calm — the tell is State.OOMKilled=true on a running container (445/448 fleet-wide), a pm2 list restart count in the double digits, and dmesg lines naming node /usr/src/a (the 15-char truncation of the worker, not the engine). Every such kill drops an in-flight run. Note worker.yml sets NODE_OPTIONS per tag but never sets SANDBOX_MEMORY_LIMIT, so the engine defaults to a ceiling larger than the whole container. Measured Aug 2026: fixing the flow-bundle leak did not reduce the kill rate at all — 484 containers went from ~1,450 restarts/hr on 0.87.0 to ~1,700/hr at +15 min and ~1,620/hr at +31 min on the fixed 0.88.0, a flat line. Sampled heaps stayed at 128 MB mean / 316 MB max while the kernel kept killing at ~1.01 GB anon-rss, which is the signature of a fast per-run spike, not accumulation (a leak raises the sampled mean; this doesn't). So do not assume a proven leak explains the OOMs — the over-commit above is the standing suspect and was never corrected. Verify any fix by soaking and comparing restart rate, never by looking at a freshly-restarted fleet, which always looks healthy.flowBundleStore.publish holds three full copies of a flow's compiled code at once, across an RPC await — this is what OOM-kills cloud workers. Confirmed Aug 2026 by snapshotting a worker caught mid-spike: 702 MB of heap against a 768 MB ceiling, of which 381.3 MB was 16 byte-identical ~23.8 MB compiledJs strings and 199.1 MB was JSArrayBufferData. publish reads every code step concurrently (Promise.all over flowSteps.code), builds manifest, then Buffer.from(JSON.stringify(manifest)) — so the step strings, the serialized string, and the Buffer are all live together. Worse, it then awaits prepareFlowBundleUpload, and the suspended async function's register file pins all three for the whole round trip; the retainer path runs through a 60 s socket-ack TimersList to element:N → property:compiledJs. One flow with ~16 heavy code steps therefore peaks near 900 MB in a single operation. Note the ordering bug too: the API can answer skip, but we build the entire manifest before asking. This is a per-run spike, not a leak — sampled heaps sit at 128 MB mean while kills happen at ~1.01 GB, so a snapshot of a randomly-chosen worker shows nothing; you have to catch one above ~430 MB RSS. Catch it below ~600 MB: raising the cgroup does not raise --max-old-space-size, so serializing a bigger heap kills the process mid-snapshot (observed).cacheState value — the cache directory is shared by every worker container on the host. worker.yml mounts one /root/cache13 into all ~28 containers, so cache.json is cross-process state, not per-process state. engine-installer used to write ENGINE_CACHE_ID = nanoid() (fresh per process) as the value and treat "value is not mine" as a miss. That only ever worked because the memo fed each process its own token back from memory; the moment reads came from disk it saw whichever container wrote last, missed on essentially every job, and re-copied main.js fleet-wide — production logged "cacheHit": false on the engine install for every single job. The rule: a cacheState value must be content-derived (a manifest, a compiled artifact, a version) so any process can validate it. Anything that means "did I do this?" belongs in a module-level variable, not on shared disk.sandbox/src/lib/cache tree has only four module-level declarations: usedPiecesMemoryCache (unbounded in key count but boolean values, capped by the piece catalog — 1 entry on cloud, where bundles ship pieces inline), pendingRemovals (set→await→delete around a non-rejecting tryCatch), and two frozen constants. In worker.ts, cachedSandboxInfo and pollLoopLiveness are reassigned wholesale, never appended. Listener accumulation on a reused sandbox is the shape to check when touching sandbox.execute() — executeProcess/executeSocket are the same objects across runs and those closures capture that run's stdOut/stdError — and it is handled: the finally clears the timeout and removes 'rpc-notify'/'close'/'error', createRpcClient is a Proxy over emitWithAck that registers nothing, and NOTIFY_EVENT is literally 'rpc-notify' so the removal string matches. Keep that last part true if you rename the event.finally, not at the end of the try. worker.ts's per-job lockExtensionInterval clears at the end of its try block. It is safe only because both awaits before it are wrapped in tryCatch, which catches everything and never rejects; put one unwrapped await between them and every job leaks a repeating 30s timer whose closure retains the job (token included) and keeps calling extendLock forever. execute-agent-run.ts is the pattern to copy — both its intervals clear in a finally.LATEST_CACHE_VERSION only works because the runtime calls deleteStaleCache — keep that call wired, and keep it on a timer. The bump changes which directory is used (cache/v14), it does not remove the old one; deleteStaleCache reclaims every non-current version, and until Aug 2026 it had zero callers, so a bump would have stranded the whole previous cache on disk indefinitely. The trap is that reclaiming it eagerly is worse than not reclaiming it: the cache volume is shared by every worker container on the host (~28 of them) and a rolling deploy runs both versions at once, so a fresh v14 worker deleting v13 on startup rips the tree out from under still-running v13 workers mid-run. So the function refuses to delete anything until the current version directory's own mtime is older than STALE_CACHE_GRACE_MS (2h) — "this host has been on v14 long enough that the rollout is over" — and it runs from the periodic sweepActionRunCache timer in worker.ts, not from prewarm, because at boot the grace period has by definition not passed. Failing the guard only costs disk, so it is the safe direction; do not "fix" it by deleting on startup, and if you refactor the sweeper move the call rather than dropping it, since the failure is silent and only shows up as disks filling with v12/v13 leftovers.@activepieces/shared and pieces-framework can never be resident twice — that is enforced by the bundler, not by luck. bundle-piece-utils.ts always inlines @activepieces/* (the esbuild plugin only ever externalizes third-party packages) and then filters @activepieces/* out of the declared external list before writing the manifest. Verified at runtime on staging Aug 2026 by Runtime.evaluate against the live sandbox-<id> engine and a heap snapshot: 0 @activepieces/shared modules and 0 pieces-framework modules in require.cache, on both the worker process (671 modules / 5.22 MB) and the engine (1017 modules / 28.12 MB) — and each piece resolves to exactly one module, its bundle.cjs. Module keys carry the proof that the bundled copy is what loaded: they read …/pieces/@activepieces/piece-slack-0.9.5/bundle.tgz/…, reached via package.json main → src/index.mjs → import './bundle.cjs'. When auditing this, ignore string-match hits inside a probe's own source text — grepping heap strings for the package name finds your own instrumentation first.bundlePiece inlines everything by default, but if the result exceeds FAIL_BYTES (5 MB) it rebuilds with inlineAll: false, externalizing all third-party deps (bundle-piece-utils.ts) — deliberate, so every piece keeps building and the tarball stays small. The runtime bill lands elsewhere: google-sheets has done this since 0.8.4 (10 externals, growing to 17 by 0.16.1), so its bundle declares [email protected], which bun installs into the shared workspace store and the engine loads — 23.52 MB of a 28.12 MB require.cache, 83%, for one piece, while fully-inlined [email protected] contributes 1.58 MB despite being the larger piece. Two googleapis-common majors (7.0.1 and 7.2.0) sit resident together, which is exactly the duplication bundling was meant to end. Sampling 60 CDN bundles: 60/60 carry bundle.cjs, 55/60 declare zero dependencies, and the 5 that don't externalize only native/wasm-ish packages (pino, jimp, pdf-lib, undici, bufferutil, tiktoken). So "bundled" is not a uniform guarantee — check package.json.dependencies inside the tarball before assuming a piece is self-contained.package.json declares no dependencies and every require() in src/bundle.cjs is a Node built-in. Good for install size and it is what finally kills the multi-copy @activepieces/shared problem — but two pieces that both use axios/googleapis no longer share one hoisted copy, they each carry their own. Measured on staging Aug 2026 (AP_USE_CDN_FOR_BUNDLES=true, isolate mode): 9 fat bundled pieces in one flow = ~15 MB of uncompressed JS, engine at 926 MB RSS / 788 MB heapUsed against a V8 heap_size_limit of 1216 MB (--max-old-space-size=1024 from the default SANDBOX_MEMORY_LIMIT) inside a 1 GiB container already holding a 247 MB worker + 74 MB pm2 → MEMORY_LIMIT_EXCEEDED, reproducibly, in ~8 s. The identical flow succeeds with the container raised to 3 GiB. This is the deterministic reproducer for the standing over-commit above: a per-run spike, not accumulation. Sizing rule: budget the engine ceiling against container minus worker minus pm2, and treat "many fat pieces in one flow" as the spike driver now that bundles don't share.@activepieces/shared copy is zod schema construction, not data — ~40 MB per copy, and reused engines collect one per piece version. Heap-snapshotted a dedicated-worker engine at 596 MB RSS / 402 MB heap (0.88.1 beta, pre-CDN-bundles, AP_REUSE_SANDBOX), Aug 2026: after a double forced GC, 393 MB survived, and the histogram was 2.05 M anonymous closures (109 MB) + 143 MB of (object properties) arrays + 359 k system/Context scopes (18 MB) — instantiated module graphs, not retained run data (only 16 k distinct functions back those 2 M closures). Every sampled retainer path ended require.cache → @activepieces+shared@<version>/…/<dto>.js → exports → <ZodSchema>._def → get shape → refine/pipe/optional/brand closures: shared eagerly builds ~500 top-level zod DTO trees at import, and zod v4 attaches per-instance accessor/method closures (120 k get, 67 k set, 31 k validate in that one heap). The engine held 9 shared versions at once (0.37 → 0.96.2) because each piece bundle pins its own, and reuse + import() pins them forever — including 3 versions of piece-hubspot and 2 of piece-slack simultaneously, one per flow-pinned piece version, so republish/upgrade churn multiplies copies of the same piece. Budget ≈ 60–70 MB engine baseline + ~40 MB per distinct resident shared copy. Two independent fixes attack it: CDN self-contained bundles (no shared inside pieces at all, see above) and require()-based piece LRU eviction; snapshot mechanics for redoing this measurement are in the profile-worker-memory skill. Follow-up exact measurement (same process at 542 MB heap, 9 shared copies, graph-cut retained sizes): one cleanly-severable copy weighs 41.3 MB / 445 k nodes, but the other 8 each show <1 MB exclusive retained because copies are co-retained as clusters, and the pin is the ESM module map, not require.cache: deleting every require.cache entry under both .bun store roots frees only ~65 MB of 553 MB, while blocking the ESM ModuleWrap/SyntheticModule entries too frees 406 MB (engine-only floor: 147 MB, which includes main.js's own bundled shared). Retainer chain: Global handles → SyntheticModule (piece import()) → Piece object → action run closures → context:pieces_framework_1 / shared_1 (whole exports) — every action closure captures its module scope, so a piece and its framework+shared copies live and die together. Consequence: any eviction scheme must make the piece entry itself collectable (hence require()-based loading in the LRU fix — an import()ed entry can never be dropped); purging shared's cache entries alone reclaims ~nothing.WorkerJobType must be added to USER_INTERACTION_JOB_TYPES in packages/server/api/src/app/workers/job-queue/job-queue.ts. jobBroker.completeJob only publishes the engine response back to the waiting webserver for job types in that set — miss it and the caller hangs to WATCHER_SAFETY_TIMEOUT_MS (5 min) with no error. submitAndWaitForResponse has only that backstop, so any best-effort caller must additionally cap itself (Promise.race with a short timeout); the losing engine job still runs to completion, so the cap buys back user latency, not fleet capacity.No handler = the worker runs the wrong edition. The single shared system-job-queue is consumed by whichever app instance runs startWorker(), and EE handlers only register in the CLOUD/ENTERPRISE branches of the edition switch. A worker on a different edition than the instance that scheduled the job throws No handler for job <name> every tick. Seen July 2026: ~14.6k failures, ~99% chat-stale-sweep, because the worker defaulted to community (AP_EDITION unset) inside a cloud deployment — CE jobs like file-cleanup-trigger ran fine on the same worker, every EE-scheduled job failed identically. Fix on the deployment (AP_EDITION=cloud), not by registering EE handlers in CE. The count looks huge because removeOnComplete: true hides successes and removeOnFail has an age cap but no count cap.JobSchedulerJson.id is almost always undefined — never filter schedulers on it. BullMQ only populates id on the legacy keyToData path (raw name:jobId:endDate:tz:pattern zset members); for both modern job schedulers and hashed legacy repeatables it is absent, and the identity you pass to removeJobScheduler is key. removeDeprecatedJobs (helper/system-jobs/system-job.ts) filtered on !isNil(f.id), so from 0.86.x it removed nothing — then the one-time pass found the scheduler's live delayed job and job.remove() threw Job repeat:<hash>:<millis> belongs to a job scheduler and cannot be removed directly (the lua refuses when rjk is still scored in the repeat zset), and the Promise.all aborted the rest of the cleanup. Order matters: remove the scheduler first, then the orphaned delayed job removes cleanly. Use allSettled for boot-time cleanup so one stuck entry can't block every other removal, and guard deprecated-name matching with !knownJobNames.includes(name) since the match is startsWith.kamal app exec on the worker image used to leak a permanent worker (fixed)docker-entrypoint.sh never honored "$@": whatever command it was handed, it built /tmp/ecosystem.config.js from AP_CONTAINER_TYPE and ended on pm2-runtime start. Since the Dockerfile uses exec-form ENTRYPOINT, a run-time command arrives as arguments, not a replacement — so kamal app exec <cmd> against activepieces-cloud never ran <cmd>. It inherited the role's AP_CONTAINER_TYPE=WORKER, booted a full permanent worker, and returned no output. One container per host, per invocation. Fixed by an if [ "$#" -gt 0 ]; then exec "$@"; fi placed before all setup, so an exec'd command skips JWT generation and the PM2 config write; safe because normal boot passes no arguments and the only CMD in the Dockerfile belongs to HEALTHCHECK.
One kamal app exec "ls /root/codes" on 2026-07-09 left 412 orphan containers (activepieces-shared05_<N>-exec-latest-<hash>, one per worker slot across 25 hosts) still running 18 days later with RestartCount: 0, holding ~86 GB RAM fleet-wide. And they were not harmless: they came up on 0.86.1, exactly the release the fleet was running that day, so the version gate did not block them — for ~11 days they were fully eligible workers executing production jobs as untracked containers no kamal deploy could drain or upgrade. The gate is what eventually stopped them once the fleet reached 0.86.2/0.86.3, not what prevented them. At least the second occurrence; a lone 0.85.2 worker from an earlier leak was already logging at 1,860/hr on 2026-07-08.
Two structural reasons it hid for 18 days: HEALTHCHECK exits 0 unconditionally for AP_CONTAINER_TYPE=WORKER (workers have no HTTP server), so orphans always read (healthy); and the version-mismatch warning fired ~1.9M times per 12h with nothing alerting on it. "A worker container is running a release other than the deployed one" is still unalerted — that is the gap worth closing.
A second leak shape exists that the exec-latest name filter misses. Found 2026-07-27 on app host 46.224.20.109: a container Docker had auto-named gracious_clarke, up since 2026-06-09, Cmd: ["bash"] — a plain interactive docker run, not kamal app exec, so no exec-latest in the name and nothing to filter on. The entrypoint booted pm2 anyway (same root cause as above, predating the fix), giving a full production APP node — AP_CONTAINER_TYPE=APP, live Postgres and Redis, AP_EDITION=cloud — running system jobs on a 6-week-old build with 2+ days of CPU burned. Being an APP, the worker version gate never applied to it at all. Note the registry claim below still holds but the local daemon does carry a latest tag: this container's .Config.Image reads :latest, resolving to image 313cc440c95d = 0.83.0.154094fb, three releases behind, 1.97 GB, present on no other app host because the live container blocks image prune. The detection that catches every shape regardless of name is count running containers per host against the expected number — the four healthy app hosts had 9, this one had 10; shared workers 29 (28 + proxy), dedicated 4. Compare docker ps --format '{{.Image}}' against the deployed tag for the same result.
Don't go hunting a stale :latest in the registry — activepieces-cloud has no latest tag published at all (4,000 versions scanned back to 2026-05-07); exec-latest is Kamal's naming for an exec container, not a registry reference. A leaked container is simply pinned forever to whatever release was deployed the day it leaked. Cleanup is name-scoped, since live containers are …-<version>.beta: docker rm -f $(docker ps -aq --filter name=exec-latest). Leave Exited (137) containers alone — they're the kamal rollback targets. Now that the entrypoint works, app exec is for fresh-container jobs (rollback and migration commands — the --entrypoint npm hack in docs/install/configure-operate/rollback.mdx can go, inspecting what shipped in an image, checking config resolution under a role's env); use --reuse when you need the running container's live state.
Whole-fleet throughput can collapse while every worker is healthy and idle, because dispatch, not execution, is the cap. createQueueDispatcher (queue-dispatcher.ts) runs one serial runLoop per queue name: every iteration awaits tryDequeue before handing a job to the next waiter, and for EXECUTE_POLLING / RENEW_WEBHOOK that includes a synchronous Postgres round-trip in zombiePollingInterceptor → triggerSourceRepo().findOneBy({ flowVersionId }). A queue's dispatch rate is therefore 1 / iteration_latency no matter how many workers poll — sustaining ~55 polling jobs/sec needs sub-18ms iterations including that query. No metric says "dispatcher saturated"; you infer it.
Don't "just" parallelize it. getNextJob moves a job wait → active at dequeue, so the app owns it before the worker starts and it only leaves active on completion or the slow stalled-scan. #11792 (2026-03-16) added the waiter queue precisely so a job is never pulled out of Redis unless a worker is already parked waiting; before that, active ran away past total worker concurrency, and that incident is pinned by test/unit/app/workers/job-queue/active-invariant.test.ts. The two invariants are in tension — serial dispatch protects active but caps throughput. #13962 (2026-06-28) added a second independent guard (jobAssignmentTracker + releaseConnectionJobs), so the serialization is arguably now redundant; #13998 (2026-07-01) made polls dequeue concurrently again and was reverted by #14316 (2026-07-20) with no recorded reason — note the concurrent version dropped the onOrphanedJob path, which likely motivated the revert. Any future fix has to satisfy both invariants at once.
Diagnostic signature (how it was found 2026-07-25/26), all from ClickHouse default.otel_logs: worker fleet constant (uniqExact(LogAttributes['host']) on event='system.snapshot'), per-job latency flat or improving (avg(executionMs) over job.execute) so workers aren't the bottleneck, and app-side dequeues falling on the same curve as executions. The decisive one is uniform stretch: for EXECUTE_POLLING, group by hour and compute count() / uniqExact(flowVersion.id). Flat distinctTriggers with decaying fires-per-trigger (13,050 triggers steady while each went 15.2/hr → 1.05/hr over 30h) means a shared dispatcher throttling all of them equally; triggers actually dying looks the opposite — distinctTriggers drops, fires-per-trigger holds. Rule out diurnal against the same hours a week earlier first.
Three latent hazards in the same file: await dequeue(...) has no timeout, so a hang on Redis leaves loopRunning true forever and every later poll() short-circuits, parks for WAITER_TIMEOUT_MS (50s) and returns null — that queue is dead for the life of the process and only a restart clears it. tryDequeue recurses once per skipped job (deferred-failure, invalid-schema, DISCARD, delay paths), so discarded zombie repeats burn dispatcher budget without producing executions (520k dequeues vs 270k executions) and a burst drives unbounded async recursion. And concurrency: 500 on the BullMQWorker is inert — autorun: false means BullMQ's internal run loop never starts, so it reads as 500-way parallel dispatch while being 1-way.
Incident 2026-07-26. workerJobs reached 229,682 prioritized / active:1 while all ~450 containers read Up (healthy). The workers were neither crashing nor disconnected: Socket.IO stayed live, they kept taking Flow published pushes and emitting system.snapshot heartbeats, they simply stopped calling poll — only 4–5 distinct worker ids appeared in [workerRpc#poll] Poll request received out of ~450.
buildMachineInfo() is awaited inside the poll loop, once per iteration, before apiClient.poll(). probeServerPing (added 2026-07-21 in PR #14181, purely to fill a serverPingMs field on the workers page) never consumed or cancelled its fetch response body, so its 5s AbortController did not release a request queued behind a busy undici client — the loop parked before its first poll, with no HTTP client handle left in process.getActiveResourcesInfo(). Three things kept that invisible: pm2 replaces the OOM-killed process every ~4 min inside the 1g container without Docker seeing anything (oom_kill 847 / pm2 ↺ 848 against RestartCount: 0, 38.7h wedged), the health server returned an unconditional 200 {"status":"ok"}, and the poll loop exited in complete silence.
Telling it apart, fast — one ratio and one grep:
Connected to API server via Socket.IO : Starting poll loops : Polling worker started. 1:1:1 with no job.execute after = wedged inside the loop. A connect with no following Starting poll loops is the if (polling) return latch in startPollingWorkers or a hung fetchAndStoreSettings instead.[workerRpc#poll] Withholding job = the version gate, not this. Then count distinct worker ids polling against fleet size — that single number separates "workers stopped asking" from "app stopped answering".prioritized huge + active ≈ 1 is not an app-side dispatcher wedge. The dispatcher is pull-only, so zero pollers means zero dequeues by design; a poll that does arrive is served in ~22ms.Recovery on a build that predates the fix (≤ 0.86.3.9c53aeaf): ~/restart-workers.sh on the DevOps box, one ssh per host, ~20 min fleet-wide. A fresh process is the only thing that re-arms the loop and it buys ~14 hours, not a fix — 13 of 14 sampled containers were wedged again the morning after the 2026-07-26 recovery. Two signals confirm the diagnosis while it rolls: wedged containers burn the full 30s SIGTERM timeout (a 2–3s stop means idle, not wedged), and restarted ones report (healthy) within 22s before polling anything, so watch workerJobs:active climbing instead. Keep 2–3 containers unrestarted for forensics — the script already skips four.
On a build that carries the fix a wedged loop restarts itself: probeServerPing races its own timeout and cancels the response body (wrapping buildMachineInfo() in a timeout alone was not enough — a hung probe still starved the loop through its 20s error-retry path), a watchdog exits the process after POLL_LIVENESS_TIMEOUT_MS (180s) with no loop iteration, the health endpoint 503s on the same condition, and the loop logs at warn when it exits. Liveness is tracked per poll loop, so with AP_WORKER_CONCURRENCY > 1 a healthy sibling cannot mask a wedged one. Still finding wedged containers on that build means the watchdog itself is not firing — new bug, preserve a container before restarting.
The OOM loop is the wedge's supply of attempts, and canary proves it. 2026-07-27, same buggy build on both tiers: the 7 canary workers (1.0 cpu, 1g, max-old-space-size=768) had 0 pm2 restarts and cgroup oom 0 after 46h uptime, and not one was wedged — all consuming 9–12 jobs per 10 min. The shared tier differs only in cpus: 0.5, and 35 minutes after a full fleet restart 135 of 448 shared slots (30%) were already wedged again, sampled containers showing 7–11 pm2 restarts in those 35 min (~1 every 3 min) with the textbook 1:1:1 connect/start/pollstart ratio. probeServerPing does not wedge a running worker; it wedges one on startup, and only the shared tier restarts often enough to keep rolling the dice. So raising shared worker cpu (or lowering max-old-space-size) is an independent mitigation that works without shipping the code fix — and it is what makes a recovery restart hold instead of decaying within the day.
Also outstanding: add packages/server/worker to CI's test filter — worker.test.ts does not run in CI today.
EXECUTE_POLLING and RENEW_WEBHOOK are veryLow = priority 5 (packages/core/execution/src/lib/workers/job-data.ts:51); webhooks and async flows are 3, sync flows and chat agents 2, user-interaction jobs 1. BullMQ pops the lowest score first, so when worker capacity drops, schedule and polling triggers are the first thing to go silent while everything else still looks fine. That makes "my */1 schedule flow isn't triggering" the earliest symptom of a partial fleet wedge, long before anyone calls it an outage. Seen 2026-07-27: 484 containers up but active:103 against a healthy ~408, the priority-5 lane holding 13,190 jobs with 5,318 of them 15–30 min overdue; one customer's poll job moved rank 8803 → 7319 in 8 minutes, i.e. ~20–75 min per poll on a one-minute cron.
Reading the lanes: BullMQ scores are priority * 2^32 + counter, so ZCOUNT bull:workerJobs:prioritized <p>*2^32 <p+1>*2^32-1 gives each lane's depth. A huge prioritized total is not by itself the signal — check which lane. Then take a repeat job's ZRANK twice a few minutes apart; that drift is the lane's true drain rate, far below the queue's overall completion rate because higher priorities keep cutting in. Lateness for a repeat job comes from the scheduled millis in its id (repeat:<flowVersionId>:<millis>), not the timestamp field, which is when the scheduler produced it — 24h early for a daily cron. Redis is DigitalOcean-managed and its cert does not verify from the DevOps box, so the working invocation is docker exec redis redis-cli --tls --insecure -h <AP_REDIS_HOST> -p 25061 --user default -a <pw>, with creds read off any activepieces-app_* container's env. Workers carry no Redis env at all — they reach the queue only through the app over Socket.IO.
Three things that look like bugs and aren't:
flowVersionId, but the new job joins the back of the same starved lane.repeat:<oldVersionId>:<millis> stays in prioritized and fires once against a soft-deleted trigger_source.RATE_LIMIT_PRIORITY is lowest = 6, so rate-limited EXECUTE_FLOW jobs re-enter behind the polling lane and never drain while polling is chronically backed up. The 2026-07-27 pile held jobs going back 19 days. Anything demoted to priority 6 on a queue with a standing priority-5 backlog is silently dropped.Entry point: worker, the exported lifecycle object in worker.ts; worker.start(...) is wired up in the worker process main.
packages/server/worker/src/lib/worker.ts — worker lifecycle, the pollAndExecute loop, version gate, in-flight drainpackages/server/worker/src/lib/config/ — worker env vars and the cached WorkerSettingsResponsepackages/server/worker/src/lib/runtime/ — bridges worker settings to sandbox settings, primes full container memorypackages/server/sandbox/src/lib/ — the @activepieces/sandbox library: Runtime / Resolver, caches, piece install, type contractspackages/server/api/src/app/workers/machine/ — Socket.IO listeners, machine service, worker-machine and queue-metrics routespackages/server/api/src/app/workers/rpc/ — app-side RPC handlers: poll, completeJob, extendLock, flow-bundle callspackages/server/api/src/app/workers/job-queue/ — job assignment tracker and queue routingpackages/core/execution/src/lib/workers/ — WorkerProps, MachineInformation, WorkerSettingsResponse, WorkerToApiContractpackages/server/utils/src/ap-version.ts — getCurrentRelease() and versionsAreCompatible()Paths verified 2026-07-17. An earlier version pointed at packages/core/shared/src/lib/automation/workers/; it moved to packages/core/execution/src/lib/workers/.