brain/flows-execution/action-run.md
An action run executes a single piece action or code step directly, outside any flow — the unit of work behind MCP's ap_run_action and the chat ap_execute_action / ap_run_code tools. It replaces the old "temporary flow" hack: create a throwaway flow → graft one step → flowRunService.test() → poll flow_run for ≤120s → dig the step out of run.steps → best-effort delete the flow.
At this stage action runs are execution only — nothing is persisted. The caller gets the outcome in-process. Durable storage (an action_run table separate from Flow Runs, its endpoints, and an "Action runs" UI tab) lands separately.
actionRunService(log).run({ projectId, platformId, step }) resolves the piece package, then submits WorkerJobType.EXECUTE_ACTION via userInteractionWatcher.submitAndWaitForResponse — synchronous request/response, the same mechanism as property resolution and auth validation. No polling, no queued flow job, no retry. See decision Action runs dispatch as synchronous user-interaction jobs.actionOperation → actionRunStepRunner.run({ step, operation }) runs the one step against FlowExecutorContext.empty() and returns steps[step.name]. The chat tool executor (engine/src/lib/tools/index.ts) calls the same primitive.deriveActionRunOutcome maps the engine response to { status, output, logs, errorMessage }. Status is a FlowRunStatus, of which only SUCCEEDED / FAILED / TIMEOUT / INTERNAL_ERROR are reachable — an action run is synchronous, so QUEUED and RUNNING never occur, and PAUSED is explicitly rejected.high, not critical, so action runs never outrank the builder interactions a human is actively waiting on.The budget is one end-to-end deadline, not two timeouts with a margin. actionRunService stamps expiresAt = now + 120s onto the job and waits 130s. The extra 10s only covers the sandbox kill and the pubsub hop home — it is not an allowance for queueing, resolution or provisioning, because those spend the same 120s: createSandboxRuntime.execute clamps the run to what is left of expiresAt after provisioning and boot, and throws SANDBOX_EXECUTION_TIMEOUT without starting the engine if nothing is left. Do not re-shape this as "sandbox 120s + a fudge factor". That was the original bug: the watcher's clock starts at enqueue and the sandbox's at run-start, so a cold piece install (easily >10s) expired the caller first while the action kept running and writing — and the retry that invites duplicates the write. The worker keeps its own 120s cap for a deadline it cannot trust (missing from an older API, or inflated by clock skew).
Derive the clamp where the timer is armed. Every phase between the two is a hole. The only thing enforcing the run budget is the SIGKILL timer armed inside sandbox.execute; remainingTimeoutInSeconds was called before sandbox.start(), so the engine got its full remaining budget counted from a later instant and user code ran until expiresAt + bootMs. Boot is the norm, not the tail — canReuseSandbox is false for SANDBOX_PROCESS unless AP_REUSE_SANDBOX=true, so every production action run pays bind retries, two unbounded isolate execPromise calls and a 30s connect cap, comfortably past the 10s grace. The pre-boot check survives only as a fast path (skip a pointless spawn); the authoritative clamp is recomputed after the sandboxStart timed block. Note the recomputed throw lands inside the try, so it invalidates a freshly booted box instead of parking it warm — one cold boot on a rare path, accepted over restructuring boot out of the try.
Do not bound provision or the connect wait by the deadline without reworking the error taxonomy first. It looks like the obvious next step — spawnWithKill already takes timeoutMs and waitForConnection's 30s is a bare literal — but both raise a plain Error, which fails isSandboxTimeout in execute-action.ts and is rethrown, so the caller gets INTERNAL_ERROR / "the engine crashed while loading or executing the piece" — the exact misreport the next gotcha forbids, in place of an honest neverStarted. Capping the connect wait also does not bound boot: it is the last of three boot phases, behind the unbounded isolate calls. Neither buys any safety, because a setup overrun already yields neverStarted with no user code executed.
A watcher timeout maps to TIMEOUT, never INTERNAL_ERROR — reporting "the engine crashed while loading or executing the piece" when nothing ran sends the calling agent off debugging the piece instead of retrying.
TIMEOUT alone is not enough: neverStarted splits it in two. "The action ran and we lost patience" and "nothing ever executed" must never share a message — only the second is safe to retry blindly, and telling an agent to check for a write when nothing ran trains it to abandon no-ops. Two sources feed one flag: the worker sets it when it refuses to start a run whose deadline already passed (SandboxExecutionTimeoutParams.neverStarted), and on a watcher timeout actionRunService calls jobQueue.cancelAndReportNeverStarted. See the area glossary — neverStarted is a proof of no-write, sound but deliberately incomplete, not a lifecycle stage.
Removal and reporting are two jobs. cancelAndReportNeverStarted always attempts the removal, and reports from job.processedOn — never from job.getState(). An earlier shape welded them: one state allowlist gated both "destroy the job so it cannot write later" and "tell the agent nothing ran". BullMQ state is not monotonic — waiting → active → delayed → waiting is legal — so a job that already executed and wrote can sit in delayed (a worker disconnect returns it there, see Workers) and read as not-yet-started. The welded version then removed it and reported neverStarted: true, telling the agent "nothing ran and nothing was written. Safe to retry as-is." — the exact duplicate side effect the no-retry decision exists to prevent.
processedOn is the sound signal and it dominates the state allowlist, so do not keep both: moveToActive sets it in the same script as the state transition, nothing on this path clears it (moveToDelayed and moveStalledJobsToWait never touch it; only Job.retry() nulls it, and that is the explicit retry-a-failed-job API), and every terminal state is reachable only through active — so completed/failed always carry it. A state read adds a round trip inside the race window and can no longer fire independently.
Removal failure stays as the second signal, for the one case processedOn cannot cover — the job grabbed between the read and the remove. Verified against pinned bullmq 5.61.0: job.remove() throws only Job <id> could not be removed because it is locked by another worker (the removeJob script returns 0 when the lock key exists and Job.remove throws on a falsy result), and succeeds on completed or failed. A throw therefore means actively-running.
Residual, accepted — re-reviewed 2026-08, kept: processedOn comes from the getJob() snapshot, so a job dequeued and completed between the snapshot and the remove() call still reports neverStarted: true. Do not read "inside one Redis RTT" as the bound — the snapshot goes stale by however long the API event loop stalls between the getJob reply arriving and being processed — so the real precondition stack is: the job starves undequeued through the full ~130s watcher budget, a worker dequeues it exactly inside that stale window, and dispatch + sandbox + execution + completeJob all finish before removeJob executes on Redis. Cold production boots are multi-second, which keeps this unreachable there; with AP_REUSE_SANDBOX=true a few-hundred-ms GC stall with perfect timing could do it — ballpark ≤1 in 10⁷ action runs even under generous assumptions. Kept because every mitigation costs more than the exposure. The known cheap close, should this ever need to die: completeJob writes a short-TTL completion receipt keyed by requestId before moveToCompleted, and cancelAndReportNeverStarted checks it after a successful remove — "removed, no processedOn, no receipt" then proves no completion, with no BullMQ internals touched. Every other interleaving is sound: a merely-dequeued job is locked so remove() throws, and a requeued or schema-failed job carries processedOn — all of them report neverStarted: false.
The price of soundness: processedOn marks dequeued, not started — the app owns the job before the worker receives it — so a job polled and then orphaned by a disconnect reports neverStarted: false though nothing ran. Most likely during a deploy, which is exactly when the honest answer would be most useful. Accepted deliberately: a spurious "check for a write" costs the agent one lookup, a spurious "safe to retry" costs the user a duplicate write.
A code step's cache namespace is action-runs/<platformId>_<sha256(sourceCode)>, and three places must agree on it. The code cache is keyed by flowVersionId + step name only, and an action run has neither — DEFAULT_MCP_DATA plus the fixed name step_1 walked straight into gotcha: the code cache is namespaced by flowVersionId: one shared dir across tenants, and no /root/codes mount at all in isolate mode without AP_REUSE_SANDBOX=true. resolveCodeStep in execute-action.ts derives the value once via actionRunCache.namespace and hands it to all three: the CodeArtifact, provision.flowVersionId (builds the mount), and EngineConstants.flowVersionId (what the engine reads). The content hash keeps code-builder's in-dir hash check from ever mismatching and lets a repeated snippet skip bun install; the platformId is what makes the directory attributable — see decision Action-run code caches live in their own directory. The action-runs/ directory level is load-bearing: it is what separates action-run builds from flow-version builds (both platformId and flowVersionId are 21-char apIds, so length cannot), and it is the sweeper's entire scope.
The namespace carries a /, so it is guarded by assertSafeCodeNamespace, not assertSafePathSegment. The nesting cannot be hidden from the engine: in fork mode AP_BASE_CODE_DIRECTORY is the raw host codes/ path with no mount indirection, so the engine reads <codes>/<namespace>/<stepName>/index.js off the host filesystem and the namespace it holds must literally contain action-runs/. assertSafeCodeNamespace splits on /, rejects more than two segments, and runs each segment through the unchanged assertSafePathSegment — so the traversal rules that gate a value destined for a bind-mount hostPath are stated in exactly one place. stepName and platformId are still single segments and still use assertSafePathSegment directly; do not widen those. Two legal segments means a/b now passes where it used to be rejected — that is deliberate, and the residual is a namespace landing one level deeper than intended, never an escape.
The sweeper cannot reach the root of codes/ at all, which is stronger than the ar_ prefix it replaced. A prefix made classification lexical and collision-proof only because ALPHABET in core-utils/id-generator.ts excludes _: adding it would have misclassified any apId beginning ar_ (~1 in 238 000, so effectively certain at scale) and the sweeper would have silently deleted flow caches. Now a flow-version cache is only reachable if it lands inside codes/action-runs/, needing a flowVersionId equal to action-runs — which requires ALPHABET to gain -, and ID_LENGTH to go 21→11, and the ApId regex to change, together. sweep also no longer filters by name: it reads only its own directory, so anything at the root of codes/ survives however old. Pinned by tests that seed a flow-version dir, a leftover ar_-prefixed dir and a stray file at the root and assert all three survive.
Content-addressing removed the only GC this path had, so the sweep is not optional. When the namespace was the constant DEFAULT_MCP_DATA.flowVersionId, every action run compiled into one directory and a different snippet was always a hash miss — and installFn opens by rm -rf-ing the directory. That destructive rebuild was the reclamation, bounded at O(1) dirs. Making the path unique per snippet fixed the concurrency race and the missing mount by making that branch unreachable, which is exactly how the GC disappeared. actionRunCache.sweep replaces it: children of codes/action-runs/ untouched for 2h are removed, then oldest-first eviction runs while more than ACTION_RUN_CACHE_MAX_DIRS (200) survive, on a 30-minute worker-local interval. localExecutionCache.provision touches each dir's mtime before the sandbox starts (gated on isActionRunNamespace, so flow-version dirs pay nothing), which is also what makes the sweep race-free — a dir currently bind-mounted was touched under ~130s ago, far inside the TTL.
A touch that lands after the sweeper's re-stat is too late, so provision also settles removals already in flight. The mtime re-check inside removeDir only rules out removals that start before the touch; one already past its re-stat deletes the tree under a sandbox that has just accepted the directory as a cache hit, and that run dies requiring a missing index.js. removeDir publishes its rm promise into pendingRemovals in the same synchronous block that starts it — before any await, which is the whole reason the two checks are exhaustive — and installCodeStep calls actionRunCache.settlePendingRemoval after the touch, awaiting the removal and rebuilding the step when one was in flight. The guard is process-local; two worker processes sharing one mount still rely on the mtime re-check alone.
Reclamation is bounded by directory count, never by bytes, and eviction exempts an active-execution window — both are safety properties, not preferences. A count cap holds the survivor floor at N whatever the dirs weigh; a byte budget's survivor count moves with size, so at 0.5 GB each a 2 GB budget permits four survivors, fewer than the live set, and must evict into it. But the cap alone is not sufficient, and it is easy to talk yourself into believing it is: mtime is provision time and is never refreshed for the ≤120s a run lasts, so a slow action run is outranked by every fast one that started after it and "the newest 200" is not "the live ones". Reaching a live dir needs only ACTION_RUN_CACHE_MAX_DIRS distinct snippets provisioned inside one execution window — ~1.7/s, reachable at cloud scale. ACTION_RUN_CACHE_ACTIVE_WINDOW_MS (15 min) is what closes it: eviction skips any dir fresher than that, so when the tree is all-active it stays above the cap until it ages out — disk overshoots instead of a run dying, and the sweep logs activeCount so a permanently blocked eviction is not silent. Keep ACTION_RUN_CACHE_MAX_DIRS > AP_WORKER_CONCURRENCY × replicas sharing the mount (25 on the reference topology against a cap of 200) — that is now a utilization invariant, not a safety one: below it, the window blocks every eviction and the cap stops holding. See decision 000016 for why a byte budget was tried and removed.
The sweep is convergent, not coordinated, because a worker cannot lock. N sweepers over one shared ./cache is the steady state (compose replicas: 5; Helm's default rollout mounts one RWO PVC into every replica), and a worker process has no Redis, so distributedLock and system jobs are both unavailable — see workers § Gotchas. Safety is structural instead: every step is idempotent (force: true, ENOENT-tolerant, mtime re-checked immediately before rm) and eviction recomputes its target from the live readdir rather than accumulating across deletions, so two simultaneous sweepers pick the same oldest set, delete it once, and neither evicts past the cap. Do not add state that accumulates across deletions — the removed byte accounting was exactly that, and it over-evicted whenever a peer deleted a dir first. Timer jitter is unnecessary for the same reason: concurrent sweeps cost duplicated stat calls, nothing more.
Directories from every earlier layout are deliberately left to leak. Bare-sha256, mcp-flow-version-id and ar_-prefixed dirs only exist on machines that ran intermediate commits of the branch that added them — none of those layouts ever reached main, so there is nothing in the field to migrate and no reclaim path was written. The sweeper does not reclaim them: the name-sniffing branch that did had no TTL and no mtime re-check, and mcp-flow-version-id is still a live constant on main (DEFAULT_MCP_DATA.flowVersionId), so the day anything provisions code under it that branch would rm -rf it every 30 minutes. On a dev box that ran those commits, rm -rf cache/v12/codes is the cleanup.
A cache hit must prove the artifact exists — do not "simplify" the compiledArtifactPresent check out of code-builder. cacheState's memo is module-scoped with no invalidation API, and a hit returns without touching disk. Delete a step dir and the memo still reports cacheHit: true, nothing rebuilds, and the engine requires a missing index.js — failing that snippet on that worker until the process restarts. Process-local invalidation would not be enough either: the reference docker-compose.yml gives app and five worker replicas the same ./cache bind mount, so one container's sweeper deletes a directory another has memoised. The one stat per hit is the only thing that makes deletion — by the sweeper, by an operator, by a reset volume — recoverable.
EXECUTE_ACTION is not project-group routable. PROJECT_GROUP_ROUTABLE_JOB_TYPES is {EXECUTE_FLOW, EXECUTE_WEBHOOK}, so action runs land on the platform/shared queue even when the project has dedicated workers — unlike the temporary-flow path they replaced, which routed as EXECUTE_FLOW. Matters when a project's own workers are the only ones that can reach its network.
actionRunMode disables the two flow-only behaviours in piece-executor.ts: the progress reporter becomes a no-op (no flow run to stream to), and waitpoints are rejected by assertActionRunCannotSuspend as a plain Error (USER-level) so the step ends FAILED, not INTERNAL_ERROR — "this action only works inside a flow" is a usage error, not an engine bug, and must not page oncall.
createWaitpointHook must throw synchronously, from the returned function's body and not from inside an async body. Do not refactor it into a single async closure. This is why the hook is split into a sync wrapper plus submitWaitpoint. The deprecated pause() shim at packages/pieces/framework/src/lib/context/versioning.ts (buildLegacyPauseHook, kept until 2026-10-12) does context.run.createWaitpoint({...}).catch(() => process.exit(1)). If the rejection arrives as a rejected promise, that .catch attaches and kills the worker process; thrown synchronously, it propagates as a FAILED step instead. Any piece still on context.run.pause() hits this path.
FLOW-scoped store entries all collide inside a project — known, accepted. context.store and context.files are HTTP-backed services needing only internalApiUrl, engineToken and flowId. With no real flow, fromExecuteActionInput substitutes the DEFAULT_MCP_DATA sentinel flowId: 'mcp-flow-id', and createContextStore builds FLOW keys as prefix + 'flow_' + flowId + '/' + key. This is not a tenancy break: storeEntryController pins projectId from the engine token, so entries never cross a project. It is a key collision — every FLOW-scoped store.put() from every action run in a project lands in one flow_mcp-flow-id/ namespace. PROJECT-scoped entries are correct, and context.files ignores flowId, so uploads are cleanly project-scoped. Fixing it means either rejecting FLOW scope in actionRunMode (breaks pieces that store state as a side effect) or a per-run flowId (makes those writes unreachable garbage) — neither is clearly right, so it stays as-is.
EXECUTE_ACTION is in UserInteractionJobData, so its payload shape is bound by LATEST_JOB_DATA_SCHEMA_VERSION — changing it needs a job-data migration. expiresAt is the exception that proves the rule: it is optional, so an old-shaped job still parses and simply behaves as it did before. Any required addition still needs the version bump.
step is validated by a real schema (ActionRunStep), not z.custom. z.custom() with no validator accepts anything — a missing step, or 42 — which defeated tryDequeue's schema gate for this job type. The same schema is parsed in actionRunService before enqueuing, because a schema failure at dequeue becomes an UnrecoverableError and that path never publishes to the watcher: the caller would hang the whole budget and then be told the action may have written.
The old path's cleanup deleted the temp flow, and flow_run.flowId is onDelete: CASCADE — so it recorded nothing durable either, despite paying for three inserts per call.
All editions (Community, Enterprise, Cloud). MCP ap_run_action is CE; the chat tools that use it are EE.
Entry point: actionRunService, defined in action-run.service.ts.
packages/server/api/src/app/action-run/ — action-run.service.ts (run()), action-run-outcome.ts (engine response → terminal status)packages/server/api/src/app/workers/user-interaction-watcher.ts — submitAndWaitForResponse, now with an optional per-caller timeoutpackages/server/api/src/app/mcp/tools/flow-run-utils.ts — executeActionRunAction / executeActionRunCode, the rewrite that deleted the temporary-flow pathpackages/core/execution/src/lib/engine/engine-operation.ts — EngineOperationType.EXECUTE_ACTION, ExecuteActionOperationpackages/core/execution/src/lib/workers/job-data.ts — WorkerJobType.EXECUTE_ACTION, ExecuteActionJobDatapackages/server/engine/src/lib/handler/action-run-step-runner.ts — the shared single-step primitivepackages/server/engine/src/lib/operations/action.operation.ts — actionOperation.executepackages/server/engine/src/lib/handler/context/engine-constants.ts — actionRunMode, fromExecuteActionInputpackages/server/engine/src/lib/handler/piece-executor.ts — the actionRunMode guardspackages/server/worker/src/lib/execute/jobs/execute-action.ts — worker handler; 120s untrusted-deadline cap, sandbox timeout → TIMEOUTpackages/server/sandbox/src/lib/cache/action-run-cache.ts — namespace / isActionRunNamespace / touch / settlePendingRemoval / sweep; naming and sweeping share one module because both are defined by the same ACTION_RUN_CODE_DIR (from cache-paths.ts, the authority on the v12/ layout)packages/server/worker/src/lib/worker.ts — startCacheSweeper / stopCacheSweeper, the 30-minute interval that drives the sweeppackages/server/sandbox/src/lib/sandbox.ts — remainingTimeoutInSeconds, the post-provision fast path and the authoritative post-boot deadline clamp (ExecuteParams.expiresAt, optional — only jobs someone blocks on set it)