web/src/features/in-app-agent/README.md
The in-app agent is Langfuse's project-scoped assistant inside the authenticated product UI. Runs execute durably in a worker while the browser observes their persisted event stream.
ARCHITECTURE.md covers why the feature is shaped this way: the one-log /
three-derivations contract, where code lives and why, and the boundaries between
browser, web server, shared runtime, and worker.
Read it before changing how messages are represented or where logic lives.
AG-UI events are the durable transcript vocabulary. Langfuse-owned protocols coordinate persistence, watching, run lifecycle, approvals, and display projection around those events.
The browser owns interaction state and submits intent. The server owns authorization, run/message IDs, request sanitization, MCP credentials, runtime configuration, tool access, persistence, and replay.
A conversation can have one active run. The browser starts a durable run, hydrates one persisted transcript/cursor snapshot, and observes its tail. Closing the drawer detaches observation without cancelling the worker run.
schema.ts: runtime-neutral AG-UI schemas and types shared by browser, server, persistence, replay, and rendering, including Langfuse-owned human-in-the-loop wire contracts.server/agent.ts: Mastra/Bedrock/MCP runtime setup, custom tool wiring, human-in-the-loop approval gates, AG-UI event normalization, and cleanup.server/human-in-the-loop.ts: interrupt parsing and worker continuation
compatibility, plus unused request-scoped pending-approval helpers retained
until the shared-runtime cleanup.server/tools.ts: custom agent tools with strict schemas and scoped, user-visible behavior.server/persistence.ts: conversations, runs, events, canonical accumulation, replay, active-run locking, and stale-run recovery. Knows nothing about rendering.lib/display.ts: display-state recording, the render-time projection, and its
wire serialization. Web-only; used by the browser and by the web server when
it builds a conversation snapshot.server/conversationSnapshot.ts: rebuilds canonical messages and display state
from one read of the persisted event log.server/router.ts: tRPC routes for durable run start/cancel/approval, conversation snapshots and lists, and feedback.server/instrumentation.ts: optional Langfuse tracing for agent runs, prompts, events, and errors.server/sandbox/config.ts: sandbox provider selection and runtime configuration.server/sandbox/service.ts: conversation-scoped sandbox session reuse, readonly file sync, and turn-end suspension.server/sandbox/providers/*: provider adapters for local Docker and Lambda MicroVM sandboxes.server/sandbox/types.ts: runtime-neutral sandbox interface used by tools/agent.constants.ts: stable names shared across prompts, tools, persistence, and rendering.components/InAppAiAgentProvider.tsx: query integration and the React bridge
to the background session.components/ControlledInAppAgentWindow.tsx and
components/InAppAgentWindow.tsx: prop-driven rendering and the explicit
execution controls.lib/backgroundAgentClient.ts: durable run start/watch AG-UI adapter.lib/backgroundExecutionSession.ts: background transcript, cursor, run,
approval, attachment, cancel, and decision owner.The worker entrypoint at worker/src/features/in-app-agent/executeInAppAgentRun.ts
owns runtime credentials, sandbox lifecycle, event persistence, approval
continuations, and terminal run transitions.
Outside this feature folder, packages/in-app-agent-sandbox-runtime/src/* provides the shared sandbox runtime and contract types used by both the local Docker provider and the Lambda MicroVM image.
flowchart TB
Provider["InAppAiAgentProvider.tsx\nclient controller"]
Provider --> Controlled["ControlledInAppAgentWindow.tsx\nconnecting pure render components to context"]
Controlled --> Window["InAppAgentWindow.tsx\nmain UI entrypoint"]
Window --> Message["InAppAgentMessage.tsx\nmessage rendering"]
Provider -->|useSyncExternalStore| Session["backgroundExecutionSession.ts\nbackground state + actions"]
Session --> BackgroundClient["backgroundAgentClient.ts\nstart + watch adapter"]
Display["lib/display.ts\ndisplay state + projection"] -.-> Provider
Display -.-> Session
Display -.-> Snapshot["server/conversationSnapshot.ts\ncanonical + display state"]
Router --> Snapshot
BackgroundClient -->|start| Router
Session -->|hydrate| Router
BackgroundClient -->|SSE tail| Watch["app/api/in-app-agent/watch\nauthenticated watch route"]
Provider -->|tRPC| Router["server/router.ts\ntRPC routes (non-streaming)"]
Schema["schema.ts\nshared AG-UI contract"] -.-> Provider
Schema -.-> HumanLoop["server/human-in-the-loop.ts\napproval state + validation"]
Schema -.-> Router
Schema -.-> Persistence["server/persistence.ts"]
Router -->|enqueue| Worker["worker executeInAppAgentRun\ndurable run driver"]
Worker --> Persistence
Worker --> Sandbox["server/sandbox/service.ts\nsandbox lifecycle"]
Worker --> Agent["server/agent.ts\nagent runtime"]
Worker --> HumanLoop
Watch --> Persistence
Sandbox --> Persistence
Sandbox --> Providers["server/sandbox/providers/*\nDocker or Lambda MicroVM"]
Agent --> HumanLoop
Agent --> Tools["server/tools.ts\ncustom tools"]
Agent --> Tools
Agent --> Instrumentation["server/instrumentation.ts\nLangfuse telemetry"]
Router --> Persistence
startRun.BackgroundExecutionSessionController installs the persisted canonical
messages, the display state, and the cursor before attaching the watch
stream. The seed is never projected or pruned; see ARCHITECTURE.md.useSyncExternalStore; it does not mirror those facts into component state.detach(), which stops browser observation only.
Reopening hydrates and resumes from the persisted cursor.BackgroundExecutionSessionController owns the live execution facts;
persisted query data only seeds the coherent bootstrap view consumed by the
same external-store hook before a controller exists. Rendering never unions
query state with a live session snapshot.InAppAgentWindow receives an execution-UI value derived from that session.useSmoothStreamingMessages; canonical AG-UI
messages are never rewritten for animation.The provider is still a large integration controller. Do not add another background state mirror.
InAppAgentWindowHost and the drawer shell mount the provider-backed controlled
window. Presentational components must remain context-free and consume explicit
props.
Streaming publications and background session snapshots are high-frequency. Keep their subscription boundary narrow, derive status/notice values during render, and preserve stable message references between session publications.
Stop intentionally emits no new analytics event during the internal rollout: the existing run lifecycle is sufficient to diagnose correctness, and there is no product decision that a separate click event would answer yet.
Rollback is a code revert; background execution does not require a schema rollback.
QUEUE_CONSUMER_IN_APP_AGENT_RUN_QUEUE_IS_ENABLED must be on wherever turns run
in background, otherwise runs commit as QUEUED and die at queue_timeout. It
is already enabled in every Langfuse Cloud region. Check
LANGFUSE_IN_APP_AGENT_MAX_ACTIVE_RUNS_PER_ORG against execution capacity too:
its default of 20 equals full US capacity and exceeds JP and staging.
Conversation switching, detached invalidations, retry, and conversation-list run statuses remain separate project concerns.
server/sandbox/service.ts gives the agent a conversation-scoped sandbox interface with read, write, and edit plus a separate turn-end callback. It reuses an existing provider session when the stored provider/session/TTL still match, otherwise it boots a fresh session and persists the new state on the conversation.
Both sandbox providers target the same runtime contract from packages/in-app-agent-sandbox-runtime.
dangerous-docker provider starts a container from that package's Docker image and calls the runtime over http://127.0.0.1:5000 using docker exec.X-aws-proxy-auth.Provider contract:
ensureSession({ conversationId, sessionId? })syncReadonlyFiles({ sessionId, files })read, write, edit, bashsuspendSession({ sessionId })Runtime HTTP surface:
GET /healthPOST /sandboxPOST /sandbox is the narrow control surface for the current tool set: read, write, edit, and bash. Before each request, the provider rebuilds tool_calls/ from persisted non-sandbox tool calls so the runtime always sees the same readonly context regardless of provider.
Sandbox state is stored on the conversation row as providerSessionId. The configured sandbox provider is assumed to remain stable for the lifetime of the database.
Session reuse only relies on an existing live or suspended runtime instance identified by providerSessionId.
server/router.ts clears sandbox state before soft-deleting a conversation.
dangerous-docker is development-only. Worker data-retention cleanup only tears down lambda-microvm sandboxes; local Docker sandbox cleanup stays in the web process where that provider is used.
The in-app agent worker uses two run-scoped inputs when calling Langfuse MCP:
x-langfuse-in-app-agent-tool-override.The API key authenticates the request and scopes it to the project. Without an override, in-app-agent keys are restricted to MCP tools annotated with readOnlyHint: true. When the user approves a single Langfuse MCP tool call, the worker creates a JSON override naming that one unprefixed MCP registry tool and passes it to the MCP route through the request header above.
MCP registry behavior:
readOnlyHint: true.RBAC is the first gate for Langfuse MCP tools. Before a tool is exposed to the model, server/tools.ts checks the signed-in user's projectRole and isAdmin against the tool's required ProjectScope with hasProjectAccess(). That means the assistant never sees tools the user could not use manually in the product UI or APIs. Human approval is a second gate on top of RBAC for tools classified as "approval": approval can allow one execution of a tool the user already has access to, but it does not widen the user's project permissions.
Human approval is separate from the MCP tool override. Shared server/tools.ts classifies every Langfuse MCP tool in IN_APP_AGENT_LANGFUSE_MCP_TOOL_POLICIES, using unprefixed MCP registry names and either "auto" or "approval". The web in-app-agent server test imports the MCP registry's McpToolName contract and verifies both type equality and runtime registry equality, so adding a Langfuse MCP tool requires an explicit in-app agent approval classification without making MCP bootstrap depend on the in-app agent.
IN_APP_AGENT_AUTO_APPROVED_TOOL_NAMES is generated from that map by prefixing Langfuse MCP tools with langfuse_ and adding local tools such as IN_APP_AGENT_REDIRECT_TOOL_NAME; docs MCP tools are auto-approved by the langfuseDocs_ prefix. server/agent.ts marks every other tool with Mastra requireApproval: true. Mastra emits an interrupt, the browser asks the user, and the router records the decision for a durable worker continuation. server/human-in-the-loop.ts adapts Mastra's runtime interrupt payload into the Langfuse-owned tool_approval_request contract from schema.ts; the browser stores and forwards only that runtime-neutral shape.
The request-scoped pending-approval persistence helpers and their table are not used by background execution. The helpers remain temporarily for the following shared-runtime cleanup; the table remains for historical data compatibility.
Sandbox tools are separate from MCP authorization. When a sandbox provider is enabled, server/tools.ts adds local read, write, edit, and bash tools backed by the sandbox provider contract rather than the MCP registry.
https://docs.ag-ui.com/llms.txt before changing event semantics, ordering, stream handling, compaction, tools, or state.