Back to Copilotkit

Frontend Tools

showcase/shell-docs/src/content/docs/integrations/ag2/frontend-tools.mdx

1.57.04.4 KB
Original Source

<video src="https://cdn.copilotkit.ai/docs/copilotkit/images/frontend-actions-demo.mp4" className="rounded-lg shadow-xl" loop playsInline controls autoPlay muted /> <Callout> This video shows the result of npx copilotkit@latest init with the implementation section applied to it. </Callout>

What is this?

Frontend actions are powerful tools that allow your AI agents to directly interact with and update your application's user interface. Think of them as bridges that connect your agent's decision-making capabilities with your frontend's interactive elements.

<Callout type="info"> CopilotKit consumes AG-UI protocol events streamed by AG2 over{" "} <code>/chat</code>. See the{" "} <a href="https://docs.ag2.ai/latest/docs/user-guide/ag-ui/" target="_blank"> AG2 AG-UI integration docs </a> . </Callout>

When should I use this?

Frontend actions are essential when you want to create truly interactive AI applications where your agent needs to:

  • Dynamically update UI elements
  • Trigger frontend animations or transitions
  • Show alerts or notifications
  • Modify application state
  • Handle user interactions programmatically

Without frontend actions, agents are limited to just processing and returning data. By implementing frontend actions, you can create rich, interactive experiences where your agent actively drives the user interface.

Implementation

<Steps> <Step> ### Setup CopilotKit
    To use frontend actions, you'll need to setup CopilotKit first. For the sake of brevity, we won't cover it here.

    Check out our [getting started guide](/ag2/quickstart) and come back here when you're setup.
</Step>

<Step>
    ### Create a frontend action

    First, you'll need to create a frontend action using the [useFrontendTool](/reference/v1/hooks/useFrontendTool) hook. Here's a simple one to get you started
    that says hello to the user.

    ```tsx title="page.tsx"

    export function Page() {
      // ...

      // [!code highlight:13]
      useFrontendTool({
        name: "sayHello",
        description: "Say hello to the user",
        available: "remote", // optional, makes it so the action is *only* available to the AG2 backend over AG-UI
        parameters: z.object({
          name: z.string().describe("The name of the user to say hello to"),
        }),
        handler: async ({ name }) => {
          alert(`Hello, ${name}!`);
          return `Said hello to ${name}!`;
        },
      });

      // ...
    }
    ```
</Step>
<Step>
    ###  Modify your agent
    Now, we'll need to modify the agent to access these frontend actions. Open your AG2 backend file and continue from there.
</Step>
<Step>
    ### Setup your AG2 backend over AG-UI

    AG2 can call frontend actions through AG-UI tool events with a standard `/chat` endpoint:

    <Tabs groupId="language" items={['Python']} default="Python">
        <Tab value="Python">
            ```python title="agent.py"
            from fastapi import FastAPI, Header
            from fastapi.responses import StreamingResponse
            from autogen import ConversableAgent, LLMConfig
            from autogen.ag_ui import AGUIStream, RunAgentInput

            agent = ConversableAgent(
                name="assistant",
                system_message="You are a helpful assistant.",
                llm_config=LLMConfig({"model": "gpt-5.4-mini"}),
            )

            stream = AGUIStream(agent)
            app = FastAPI()

            @app.post("/chat")
            async def run_agent(
                message: RunAgentInput,
                accept: str | None = Header(None),
            ):
                return StreamingResponse(
                    stream.dispatch(message, accept=accept),
                    media_type=accept or "text/event-stream",
                )
            ```
        </Tab>
    </Tabs>

    That's it. Your AG2 backend can now receive frontend action tool definitions emitted by CopilotKit through AG-UI.
</Step>
<Step>
    ### Give it a try
    You've now given your agent the ability to directly call any CopilotActions you've defined. These actions will be available as tools to the agent where they can be used as needed.
</Step>
</Steps>