ContextQMD
Back to Trigger

Tool result auditing

docs/ai-chat/patterns/tool-result-auditing.mdx

4.5.36.7 KB
Original Source

When a chat agent uses tools (especially human-in-the-loop tools that wait on addToolOutput from the frontend), you often need to fire side effects exactly once per resolved tool call:

  • Audit logs — record every tool result for compliance.
  • Billing — charge per tool invocation.
  • Notifications — alert downstream systems when a specific tool resolves.
  • Search-index updates — reflect tool outputs into a derived store.

The naive approach — "log every tool part you see" — over-counts. The same assistant message gets re-shown across re-renders, replays, and retries. You want a function of the form "is this tool result one I haven't already logged?" That's exactly what chat.history.extractNewToolResults returns.

The pattern

ts
import { chat } from "@trigger.dev/sdk/ai";
import { auditLog } from "@/lib/audit";

export const myChat = chat.agent({
  id: "my-chat",
  hydrateMessages: async ({ chatId, incomingMessages }) => {
    for (const msg of incomingMessages) {
      for (const r of chat.history.extractNewToolResults(msg)) {
        await auditLog.record({
          chatId,
          toolCallId: r.toolCallId,
          toolName: r.toolName,
          output: r.output,
          errorText: r.errorText,
        });
      }
    }
    return await db.getMessages(chatId);
  },
  run: async ({ messages, signal }) => {
    return streamText({ model: anthropic("claude-sonnet-4-5"), messages, abortSignal: signal });
  },
});

The hook fires per turn. incomingMessages is the new wire message (0-or-1-length, see v4.5 wire format change). For each new tool result on that message, write one audit row. Then return the canonical chain from your DB.

extractNewToolResults compares the message against the current chat.history chain and returns only tool parts whose toolCallId is not already resolved. That's what makes the call exactly-once:

  • A re-emitted message (same id, same toolCallId) returns [] — no duplicate log.
  • A genuinely new tool result on a known assistant message returns just the new ones.
  • A first-time tool result returns the full set.

Why hydrateMessages is the right hook

The pattern works in any pre-merge callback, but hydrateMessages is the canonical spot for two reasons:

  1. It fires before the runtime merges the incoming message into the accumulator. Once merged, the tool results are already on the chain, and extractNewToolResults returns [] for them.
  2. It always fires per turn — including HITL turns where the user resolved a tool with addToolOutput, which is the highest-volume audit event in most apps.

By the time onTurnComplete fires, the chain already contains responseMessage, so calling extractNewToolResults(responseMessage) there returns []. Don't put audit logging there for the resolution path.

Without hydrateMessages — onTurnComplete for self-emitted tool calls

If you don't use hydrateMessages, the runtime's snapshot+replay path handles persistence. You can still audit the agent's own tool executions in onTurnComplete — but compare against the prior message rather than the just-emitted one:

ts
onTurnComplete: async ({ chatId, newUIMessages }) => {
  // The assistant message from this turn is in newUIMessages.
  for (const msg of newUIMessages) {
    if (msg.role !== "assistant") continue;
    for (const part of msg.parts) {
      if (
        typeof part.type === "string" &&
        part.type.startsWith("tool-") &&
        ((part as any).state === "output-available" ||
         (part as any).state === "output-error")
      ) {
        await auditLog.record({
          chatId,
          toolCallId: (part as any).toolCallId,
          toolName: (part as any).type.slice("tool-".length),
          output: (part as any).output,
          errorText: (part as any).errorText,
        });
      }
    }
  }
},

newUIMessages is just the messages this turn produced — no prior-chain noise. Each tool part shows up exactly once.

This works for tools the agent itself calls (no HITL pause). For HITL flows where the user resolves a tool with addToolOutput, the resolution arrives on the next turn's wire message, not in newUIMessages of the resolving turn — use hydrateMessages for those.

Idempotency at the storage layer

Even with extractNewToolResults, transient failures (e.g. an audit-log POST that times out and is retried) can produce duplicates. Make the audit-log writer idempotent on toolCallId:

ts
await auditLog.upsert({
  where: { toolCallId: r.toolCallId },
  create: { /* ... */ },
  update: { /* timestamp, retry count, etc. */ },
});

toolCallId is unique per tool invocation (assigned by the AI SDK when the model emits the tool call) and stable across retries — perfect for an idempotency key.

What extractNewToolResults returns

ts
type ChatNewToolResult = {
  toolCallId: string;
  toolName: string;
  output: unknown;      // The tool's return value (carries the resolved value; in output-error state see errorText)
  errorText?: string;   // Set iff the part is in output-error state
};

Tool parts in input-available state (the model called the tool but it hasn't resolved yet) are not returned — only resolved results count.

Combining with HITL

Human-in-the-loop tools pause the turn waiting for addToolOutput from the frontend. When the user submits, the wire message carries an updated assistant message with the tool now in output-available state. extractNewToolResults against that message returns the just-resolved tool — exactly one audit row per user resolution:

ts
hydrateMessages: async ({ chatId, incomingMessages }) => {
  for (const msg of incomingMessages) {
    for (const r of chat.history.extractNewToolResults(msg)) {
      // Fires once per ask_user / approval / similar resolution
      await auditLog.record({ chatId, /* ... */ });
    }
  }
  return await db.getMessages(chatId);
}

This is the original motivator for the helper — see the HITL pattern's net-new-tool-result section.

See also