Feature: Durable Background Execution + Human-in-the-Loop (HITL)

Generated on: 2026-06-23 Status: Draft Owner: Engineering Team Related PRs: #13633 (HITL v2); Epic LE-1437 Companion document: ../../CZL/HITL_FEATURE_OVERVIEW.md — the engineering deep-dive (8-stage code walk-through), and ../../CZL/HITL_STATUS.md — the status/decision summary.

Table of Contents

1. Overview
2. Ubiquitous Language Glossary
3. Domain Model
4. Behavior Specifications
5. Architecture Decision Records
6. Technical Specification
7. Observability
8. Deployment & Rollback
9. Architecture Diagrams

1. Overview

Summary

Human-in-the-Loop (HITL) lets a flow pause mid-run to ask a human to approve, reject, or supply input, then resume exactly where it stopped. The pause is durable: it is checkpointed to the database, survives a process restart, and resumes without re-running or re-billing already-completed work. The paused/resumed run is fully observable as a single backend trace, with the approval step and the terminal output recorded as real spans.

Business Context

Agentic flows take consequential actions (calling tools, sending requests, spending tokens). Teams need a control point where a human gates a risky step before it executes, without losing the run if the browser closes or the server restarts. HITL turns a flow into a long-lived, resumable workflow rather than a single fire-and-forget request.

Bounded Context

Flow Execution — Durable Background Jobs & Graph Checkpointing.

Related Contexts

• Graph Engine (Customer-Supplier): raises and restores the pause at layer boundaries.
• Background Execution / Job Service (Partnership): persists job status and serves the single-flight resume.
• Tracing (Conformist): records the run — including the gate and the terminal output — as spans.
• Frontend (AG-UI) (Customer-Supplier): reattaches to the live stream and renders the decision surfaces.

Default-off guarantee

The pause probe is a no-op unless a component requests it, so normal flows are byte-for-byte unaffected: no checkpointing, no extra spans, no status churn.

2. Ubiquitous Language Glossary

3. Domain Model

3.1 Aggregates

Durable Run Aggregate

• Root Entity: the background Job (job_id)
• Entities: the Graph (with its run_id), the Checkpoint, the Pending Human Request
• Value Objects: JobStatus, Decision (action_id + values), request_id ({node_id}:{run_id}; agent tool approvals append the LangGraph interrupt id — {node_id}:{run_id}:{interrupt_id} — so each approval in one run is individually addressable and a stale resume for approval N is rejected during approval N+1)
• Invariants:
  • A pause must persist a checkpoint before the run yields (no lost state).
  • Resume must be single-flight: a decision is applied exactly once.
  • Resume must not re-execute already-built vertices (no re-billing, no re-fired tools); dead branches stay dead.
  • The whole run (pre-pause + post-resume) must trace into one run_id.

3.2 Domain Events

4. Behavior Specifications

Feature: A flow pauses for a human decision and resumes durably

As a flow author I want a step to pause for human approval and survive interruptions So that risky actions are gated and no run is lost if the server restarts.

Background

• Given a flow with an agent tool (or HumanInput node) that can request approval
• And durable background execution is enabled

Scenario: Agent tool pauses for approval

• Given the agent decides to call a gated tool
• When the run reaches the tool call
• Then the graph checkpoints and the job becomes SUSPENDED
• And a human_input_required event surfaces an Approve/Reject card

Scenario: Resume survives a process restart

• Given a job is SUSPENDED with a written checkpoint
• When the backend process restarts and the user approves
• Then the graph is rebuilt from the checkpoint and continues past the pause

Scenario: Resume is not a re-run

• Given a paused run with already-built vertices
• When the run resumes
• Then built vertices are restored, not re-executed (no LLM re-billing, no re-fired tools)
• And branches killed before the pause stay dead

Scenario: HITL inside a nested flow is rejected

• Given a flow whose Run Flow / Sub Flow / flow-as-tool target contains a connected Human Input node (or an approval-gated agent tool)
• When the nested flow is executed
• Then the run fails with a clear error ("cannot run as a nested flow… move the approval to the parent flow") instead of silently not pausing
• And a target flow with an unwired Human Input (no downstream consumer) still runs — it is skipped at runtime, not blocking

Scenario: Two HITL nodes in sequence

• Given a flow with two HumanInput nodes where the first gates the path to the second
• When the first is approved, the second pauses, and the second is then approved
• Then the run finalizes once — no loop back to the first node
• And only the chosen branch of each node runs (the first node's non-chosen branches stay dead across the second pause's resume, not just within the build that answered them)

Scenario: Single-flight resume

• Given two resume requests for the same SUSPENDED job
• When both arrive
• Then exactly one claims the SUSPENDED → IN_PROGRESS transition; the other is rejected (NOT_RESUMABLE)

Scenario: The resumed run is one complete backend trace

• Given a run that paused and resumed
• When the resumed run finishes
• Then a single trace holds Chat Input → Human In The Loop — {decision} → … → Chat Output
• And the trace panel shows the gate + terminal output after a full page refresh, with no client-side persistence

Scenario: A decision applied after timeout is rerouted, not lost

• Given a pending request that has a timeout policy
• When the decision arrives after the deadline
• Then reroute_decision_on_timeout resolves the effective decision deterministically

5. Architecture Decision Records

ADR-001: The pause is a signal, not a failure

Status: Accepted

Context

A component needing human input must stop the graph without corrupting state or marking the run failed.

Decision

Model the pause as a dedicated exception (GraphPausedException) raised at a layer boundary, after the graph snapshots itself and writes a checkpoint. The build seam catches it and emits a non-terminal human_input_required event — the stream ends without on_end, so the run is "waiting", not "done".

Consequences

• Benefits: clean separation of "waiting" from "failed"; resumable by construction; default-off (no probe → no-op).
• Trade-offs: every layer boundary consults a pause probe (cheap; skipped entirely when no pause is pending).

ADR-002: Resume is single-flight and rolls back

Status: Accepted

Context

A decision must apply exactly once even under duplicate clicks, retries, or concurrent callers.

Decision

Claim the job with an atomic SUSPENDED → IN_PROGRESS transition (claim_suspended_for_resume); on failure during resume, roll the status back so the decision is never silently consumed.

Consequences

• Benefits: exactly-once decisions; safe retries; no lost approvals.
• Trade-offs: a stale/duplicate resume returns NOT_RESUMABLE (handled by the UI).

ADR-003: Resume ≠ re-run

Status: Accepted

Context

Re-executing completed vertices on resume would re-bill LLMs, re-fire tools, and could revive dead branches.

Decision

Restore built vertices from the checkpoint; re-run only the paused vertex's non-input predecessors whose dropped output an unbuilt consumer will actually read (_rerun_non_input_predecessors / _unbuild_needed_dropped_producers). A producer behind a still-built, round-tripped consumer is left alone — re-running it is wasted work and, for a side-effecting node (an Agent re-bills its LLM and re-emits its message), surfaces as duplicate outputs on every later resume. Inputs (e.g. Chat Input) are never re-executed.

Consequences

• Benefits: no double billing, no duplicate side effects, deterministic continuation.
• Trade-offs: the resume re-runs a minimal predecessor set; the terminal output is produced fresh on resume (see ADR-004).

ADR-004: The whole run is one durable backend trace (gate + Chat Output as spans)

Status: Accepted (2026-06-23)

Context

Originally, resumed runs lost trace data in the backend: the Chat Output span was missing and the Human In The Loop step existed only as a frontend-injected node persisted in localStorage — invisible to other devices/users and incomplete in the DB. Root cause: the resume path never initialized tracing (trace_context_var was unset, so post-pause vertices skipped add_trace), and the initial run traced under a throwaway uuid instead of the run_id, splitting the run into an orphan trace.

Decision

1. On resume, call graph.initialize_run() on the checkpoint's run_id so resumed vertices trace into the same trace as the pre-pause run, and restore flow_name for the trace title.
2. Pin the caller's run_id in build_graph_from_data before initialize_run (forwarded by create_graph), so the initial run traces into graph_run_id — not a fresh uuid.
3. Record the resolved gate as a real span via TracingService.record_event_span ("Human In The Loop — Approved/Rejected").
4. Remove the frontend localStorage persistence; TraceDetailView reads the gate + output from the backend trace and only synthesizes a gate for the live window, deduped by name.

Consequences

• Benefits: one complete trace (Chat Input → gate → … → Chat Output) durable across refresh, devices, and users; no client-side band-aid.
• Trade-offs: the resume re-initializes tracing once (a no-op when tracing is disabled).
• Impact on Product: the trace panel is now the source of truth for what a HITL run did.

ADR-005: Terminal span flush race fixed

Status: Accepted

Context

Component spans are recorded asynchronously via a worker queue. At end-of-run the worker was cancelled while the terminal component's end event was still enqueued, dropping the Chat Output span in ~7 of 8 runs.

Decision

Drain the queue inline after cancelling the worker (service.py::_stop), and force-complete any started-but-unended span at flush (native.py::_finalize_pending_spans).

Consequences

• Benefits: no executed component is silently dropped from a trace.
• Trade-offs: a tiny synchronous drain at shutdown (bounded by queue size).

6. Technical Specification

6.1 Pause → suspend (initial run)

• The graph consults a pause probe at each layer boundary; on a pending pause it snapshots itself, writes a checkpoint row (always-writable schema), and raises GraphPausedException.
• The build seam (api/build.py) catches it, persists the human-input card to history, emits human_input_required, and ends the stream without on_end.
• The runner marks the job SUSPENDED (services/background_execution/runner.py).

6.2 Resume (single-flight)

• POST /api/v2/workflows/{job_id}/resume with {request_id, decision}.
• The route re-enforces FlowAction.EXECUTE (ensure_resume_execute_permission) before applying the decision — job ownership alone is not enough once shared access has been revoked.
• claim_suspended_for_resume performs the atomic status claim; failure → NOT_RESUMABLE.
• The answered card is stamped via mark_card_answered using the card_message_id snapshotted before the continuation is enqueued, and only when the card's request_id matches — so a first decision can never resolve a second pause's card.
• The gated vertex is located with request_id_targets_vertex (lfx.run.hitl), which accepts both the node shape and the nonce-suffixed tool-approval shape.
• build_resumed_graph_and_get_order (api/build.py):
  pythonCopy

  graph = LfxGraph.resume_from_checkpoint(checkpoint, checkpoint_store=store)
  graph.flow_name = graph.flow_name or flow_name
  await graph.initialize_run()                      # re-init tracing on the original run_id
  decision = reroute_decision_on_timeout(pending, resume["decision"])
  graph.human_input_decisions = {resume["request_id"]: decision}
  action_id = str((decision or {}).get("action_id", ""))
  gate_label = "Rejected" if "reject" in action_id.lower() else "Approved"
  graph.tracing_service.record_event_span(
      span_id=f"hitl-{resume['request_id']}",
      name=f"Human In The Loop — {gate_label}",
      outputs={"decision": action_id},
  )
  # restore built vertices; re-run only non-input predecessors of the gated vertex
  

6.3 Trace continuity (initial run)

• build_graph_from_data (api/utils/flow_utils.py) pins the run id before tracing starts:
  pythonCopy

  if (caller_run_id := kwargs.get("run_id")) is not None:
      graph.set_run_id(caller_run_id)
  await graph.initialize_run()
  

• create_graph forwards run_id=str(job_id) if job_id is not None else run_id to both build_graph_from_data and build_graph_from_db.

6.4 Gate span recording

• TracingService.record_event_span(span_id, name, outputs, span_type="chain") writes a standalone start+end span into the active trace for every ready tracer; deactivated/contextless calls are no-ops.

6.5 Frontend (reads backend spans)

• useGetTraceQuery → GET /api/v1/monitor/traces/{id} returns the span tree.
• TraceDetailView.tsx: renders the backend gate span; backendHasGate dedup prevents a duplicate synthetic node; executedOutputSpans bridges Chat Output from live flowPool only during the resume window (deduped by name).
• hitlStore.ts: in-memory only (pending slot); localStorage/persist removed.

6.6 lfx serve durable mode (LE-1695)

Bare lfx serve gains the same background + HITL contract without a database, backed by the single-node SQLite substrate:

• Substrate (lfx/services/durable/): SqliteDurableJobStore (job rows, gap-free per-job event log, control signals, atomic claim_suspended_for_resume) and SqliteCheckpointStore (the existing CheckpointStore ABC on the same DB file). Stdlib SQLite, WAL, async via asyncio.to_thread; one crash-safe file per deployment.
• Host (lfx/cli/serve_durable.py): DurableServeWorkflowHost extends ServeWorkflowHost with supports_background = True; submit_background creates the job row and spawns the runner; get_job_status rebuilds a completed run's WorkflowExecutionResponse from stored output_events; stop_job records a durable STOP signal and cancels the local task.
• Runner (_drive): runs graph.process() with checkpointing attached; GraphPausedException → job SUSPENDED + pending request persisted in job metadata; completion → terminal ComponentOutputs persisted as the job result; failure → FAILED with the error recorded.
• Resume: POST /api/v2/workflows/{job_id}/resume (same request/response contract as the backend, incl. 404/409 semantics and the 422 INVALID_DECISION guard against actions outside allowed_decisions) claims single-flight, restores from the checkpoint — in-process or after a restart — re-applies the caller identity, injects the decision, un-builds the gated vertex plus its opaque-dropped producers, and re-drives. Agent pause blobs persist in the same SQLite file via set_default_checkpoint_store, so tool-approval interrupts also survive a restart. GET /api/v2/workflows/{job_id}/pending exposes the pending request.
• Events (SSE): GET /api/v2/workflows/{job_id}/events re-attaches to a background run — the links.events URL advertised on every job response. It replays the durable event log after Last-Event-ID (each frame keyed by its seq), then tails until the run ends or suspends; a HITL pause replays and closes the stream, so the client reconnects with Last-Event-ID after resuming. Unknown job → 404.
• Opt-in: LFX_SERVE_DURABLE_DB=<path>. Unset → serve stays stateless and mode: background keeps its 422; workers in multi-worker mode inherit the env var and share the DB file (a stop cancels the run only in the worker executing it).
• Proof: src/lfx/tests/unit/cli/test_serve_durable.py — background echo completes with the sync-shaped response; a HITL flow suspends, exposes its pending request, resumes only the approved branch, and resumes to completion on a fresh app instance over the same DB file (restart equivalence). The {job_id}/events SSE stream replays a completed run, surfaces the human_input_request on a suspend, and — reconnected with Last-Event-ID after a resume — delivers only the post-resume human_input_decision + end frames.

6.7 Component map (quick index)

7. Observability

• Trace panel (Flow Activity): the run renders as a span tree — Chat Input → tools → Agent → Chat Output — and, for a gated run, a Human In The Loop — {decision} span. Read via GET /api/v1/monitor/traces/{id}.
• Job status: SUSPENDED / IN_PROGRESS / COMPLETED / FAILED exposed through the workflows API; pending requests via GET /api/v2/workflows/pending?flow_id=.
• Validated end-to-end (2026-06-23): a driven pause→approve→resume yields one trace of 21 spans (Chat Input → Human In The Loop — Approved → Calculator → URL → Agent → fetch_content → … → Chat Output), confirmed in the DB, in the /monitor/traces/{id} payload, and in the trace panel after a page refresh.

8. Deployment & Rollback

• Default-off: no component requests a pause → no checkpointing, no extra spans, no status churn. Safe to ship.
• Schema: the checkpoint and span/trace tables back the durability + observability; no destructive migrations are part of this feature.
• Run modes: works under the durable background path (POST /api/v2/workflows, mode: background) with resume via POST /api/v2/workflows/{job_id}/resume. On bare lfx serve, the same contract is opt-in via LFX_SERVE_DURABLE_DB=<path> (SQLite substrate, no database service); unset keeps serve stateless.
• Rollback: reverting the trace-continuity change restores the prior behavior (Chat Output/gate absent from resumed backend traces) but does not affect pause/resume correctness; reverting the pause/resume layer disables HITL entirely (flows run straight through).

9. Architecture Diagrams

9.1 Context Diagram (Level 1)

C4Context
  title System Context — Durable HITL
  Person(user, "User", "Approves/rejects a gated step")
  System(langflow, "Langflow", "Flow execution platform")
  SystemDb(db, "Database", "checkpoint, job status, trace/span")
  Rel(user, langflow, "Runs flow; answers Approve/Reject")
  Rel(langflow, db, "Persists checkpoint + status + spans")
  Rel(langflow, user, "Surfaces card / trace gate")

9.2 Sequence — pause → suspend → resume

sequenceDiagram
  participant U as User
  participant API as Workflows API
  participant G as Graph Engine
  participant DB as Database
  participant T as Tracing

  U->>API: POST /v2/workflows (background)
  API->>G: build + run (run_id pinned)
  G->>T: trace Chat Input … Agent
  G->>DB: write checkpoint (layer boundary)
  G-->>API: GraphPausedException → human_input_required
  API->>DB: job SUSPENDED
  API-->>U: Approve / Reject card
  U->>API: POST /v2/workflows/{job}/resume {decision}
  API->>DB: claim SUSPENDED→IN_PROGRESS (single-flight)
  API->>G: resume_from_checkpoint + initialize_run(run_id)
  G->>T: record "Human In The Loop — Approved" span
  G->>T: trace re-run predecessors … Chat Output
  G->>DB: flush spans (one trace) + job COMPLETED
  API-->>U: result; trace panel shows full run

9.3 Trace continuity (why one trace)

graph TB
    A[POST /v2/workflows] --> B[create_graph]
    B --> C[build_graph_from_data]
    C --> D{run_id pinned?}
    D -->|yes set_run_id job_id| E[initialize_run → trace = graph_run_id]
    D -->|no - old bug| F[fresh uuid → orphan trace]
    E --> G[pause → checkpoint stores run_id]
    G --> H[resume_from_checkpoint set_run_id run_id]
    H --> I[initialize_run → SAME trace]
    I --> J[record gate span + Chat Output]
    J --> K[(One complete trace)]

9.4 Frontend read path (no localStorage)

graph LR
    P[Trace panel] --> Q[useGetTraceQuery]
    Q --> R[GET /api/v1/monitor/traces/id]
    R --> S[span tree incl. gate + Chat Output]
    S --> T[TraceDetailView renders backend spans]
    T --> U{backendHasGate?}
    U -->|yes| V[use backend gate - no synth]
    U -->|live window| W[synth gate / bridge Chat Output from flowPool]