examples/canvas/llamaindex-composio/README.md
Welcome to the Fullstack Agents hackathon! This starter gives you a complete AI-powered canvas application with real-world integrations. Utilizing LlamaIndex, Composio, and CopilotKit.
This is a starter template for building AI-powered canvas applications using LlamaIndex, CopilotKit, and Composio. It provides a modern Next.js application with an integrated LlamaIndex agent that manages a visual canvas of interactive cards with real-time AI synchronization and external tool integrations (Google Sheets, for this example) through Composio.
This is an example application that we built to help you get started quickly. Everything you see can be customized, replaced, augmented or built upon.
https://github.com/user-attachments/assets/2a4ec718-b83b-4968-9cbe-7c1fe082e958
LlamaIndex is a framework for building generative AI applications, in particular Document Agents, i.e. agents that process unstructured data like PDFs, PowerPoints, Word files and more. The core framework has adapters for loading and storing data, while the Workflows framework provides a way to build an agent or multi-agent system and control how data moves around. Both frameworks can make use of LlamaCloud, an enterprise offering that provides RAG and structured data extraction as a service.
Composio is the fastest way to enable your AI agents to take real-world actionsβwithout dealing with individual API integrations, authentication flows, or complex tool formatting. It provides access 3000+ tools out of the box across popular apps like Slack, GitHub, Notion, and more.
CopilotKit connects your app's logic, state, and user context to the AI agents that deliver the animated and interactive part of your app experience β across both embedded UIs and fully headless interfaces. It gives you the tools to build, deploy, and monitor AI-assisted features that feel intuitive, helpful, and deeply integrated.
This repository is designed to help you hit the ground running for the hackathon. Use it as a foundation for your project, a source of inspiration, or simply as a quick way to get started. The following steps will guide you through setting up the project locally.
In case you get stuck, we highly recommend checking out the documentation.
Plug-in these resources to let coding agents help you and our team!
Before getting started, you'll need to the following.
agent/.env) (platform.openai.com/api-keys)agent/.env) (app.composio.dev/developers)[!WARNING] If you run into problems getting started, make sure you have all the Prerequisites installed, or else it can fail.
Install dependencies using your preferred package manager:
This will install both your Node and Python dependencies (using uv).
# Using pnpm (recommended)
pnpm install
# Using npm
npm install
# Using yarn
yarn install
# Using bun
bun install
Note: This repository ignores lock files (package-lock.json, yarn.lock, pnpm-lock.yaml, bun.lockb) to avoid conflicts between different package managers. Each developer should generate their own lock file using their preferred package manager. After that, make sure to delete it from the
.gitignore.
Setup Googlesheets Integration
Navigate to https://app.composio.dev/developers, setup a Google Sheet integration and grab an API key.
For the next step you'll need a Composio API key, auth config ID, and user ID.
<table> <tbody> <tr></tr> <tr> <td> <details> <summary> <strong>Show me how</strong></summary>https://github.com/user-attachments/assets/159cec99-7ba2-4ee2-83dd-2d86e51afea1
</details> </td> </tr> </tbody> </table>Set up your environment variables:
There are two .env files to configure:
Copy agent/.env.example to agent/.env:
# OpenAI API key
OPENAI_API_KEY=""
# Composio secrets
COMPOSIO_API_KEY=""
COMPOSIO_GOOGLESHEETS_AUTH_CONFIG_ID=""
COMPOSIO_USER_ID="default"
Note: The OpenAI API key is required for the LlamaIndex agent to function
Copy .env.local.example to .env in the root directory:
# .env.local
COPILOT_CLOUD_PUBLIC_API_KEY="" # optional (for CopilotKit Cloud features)
Start the development server:
# Using pnpm
pnpm dev
# Using npm
npm run dev
# Using yarn
yarn dev
# Using bun
bun run dev
This will start both the UI and agent servers concurrently.
You're done! β
Open http://localhost:3000 to use the starter and try it out!
<table> <tbody> <tr></tr> <tr> <td> <details> <summary> <strong>Using the canvas starter</strong></summary>Once the application is running, you can:
Create Cards: Use the "New Item" button or ask the AI to create cards
Edit Cards: Click on any field to edit directly, or ask the AI
Sync with Google Sheets: Use the Google Sheets button or ask the AI
Execute Plans: Give the AI multi-step instructions
View JSON: Toggle between the visual canvas and JSON view using the button at the bottom
The following scripts can also be run using your preferred package manager:
dev - Starts both UI and agent servers in development modedev:debug - Starts development servers with debug logging enableddev:ui - Starts only the Next.js UI serverdev:agent - Starts only the LlamaIndex agent serverinstall:agent - Installs Python dependencies for the agentbuild - Builds the Next.js application for productionstart - Starts the production serverlint - Runs ESLint for code lintinggraph TB
subgraph "Frontend (Next.js)"
UI[Canvas UI
page.tsx]
Actions[Frontend Actions
useCopilotAction]
State[State Management
useCoAgent]
Chat[CopilotChat]
SheetsBtn[Google Sheets
Menu]
end
subgraph "Backend (Python)"
Agent[LlamaIndex Agent
agent.py]
Tools[Backend Tools
- set_plan
- update_plan_progress
- complete_plan]
SheetsTools[Sheets Tools
- sheets_sync_all
- sheets_create_new
- sheets_get_url]
AgentState[Workflow Context
State Management]
Model[LLM
GPT-4o]
end
subgraph "External Services"
Composio[Composio API
Tool Integration]
GSheets[Google Sheets
API]
end
subgraph "Communication"
Runtime[CopilotKit Runtime
:9000]
end
UI <--> State
SheetsBtn --> UI
State <--> Runtime
Chat <--> Runtime
Actions --> Runtime
Runtime <--> Agent
Agent --> Tools
Agent --> SheetsTools
SheetsTools --> Composio
Composio --> GSheets
Agent --> AgentState
Agent --> Model
style UI text-decoration:none,fill:#e1f5fe
style Agent text-decoration:none,fill:#fff3e0
style Runtime text-decoration:none,fill:#f3e5f5,color:#111111
style Composio text-decoration:none,fill:#e8f5e9,color:#111111
style GSheets text-decoration:none,fill:#fff9c4,color:#111111
click UI "https://github.com/CopilotKit/CopilotKit/blob/main/examples/canvas/llamaindex-composio/src/app/page.tsx"
click Agent "https://github.com/CopilotKit/CopilotKit/blob/main/examples/canvas/llamaindex-composio/agent/agent/agent.py"
sequenceDiagram
participant User
participant UI as Canvas UI
participant CK as CopilotKit
participant Agent as LlamaIndex Agent
participant Tools
participant Composio
participant GSheets as Google Sheets
User->>UI: Interact with canvas
UI->>CK: Update state via useCoAgent
CK->>Agent: Send state + message
Agent->>Agent: Process with GPT-4o
Agent->>Tools: Execute tools
alt Google Sheets Sync
Agent->>Composio: Execute sheets_sync_all
Composio->>GSheets: Update spreadsheet
GSheets-->>Composio: Confirm update
Composio-->>Agent: Return status
end
Tools-->>Agent: Return results
Agent->>CK: Return updated state
CK->>UI: Sync state changes
UI->>User: Display updates
Note over Agent: Maintains ground truth
Note over UI,CK: Real-time bidirectional sync
Note over Composio,GSheets: External tool integration
The main UI component is in src/app/page.tsx. It includes:
useCoAgent hook for real-time state sync with the agentuseCopilotActionuseCopilotAction with renderAndWaitForResponse for disambiguation prompts (e.g., choosing an item or card type)The agent logic is in agent/agent/agent.py. It features:
get_ag_ui_workflow_router for seamless integrationCard Field Schema
Each card type has specific fields defined in the agent:
With LlamaIndex's get_ag_ui_workflow_router, you can define backend tools.
def hello_world(name: str) -> str:
return f"Hello, {str}" # tool result
agentic_chat_router = get_ag_ui_workflow_router(
llm=OpenAI(model="gpt-4.1"),
backend_tools=[hello_world]
)
You can then render this with CopilotKit's useCopilotAction
// page.tsx
import { useCopilotAction } from "@copilotkit/core"
const Main = () => {
// ...
useCopilotAction({
name: "hello_world"
render: () => {
return <div>Called hello_world tool...</div>
}
})
// ...
}
With LlamaIndex's get_ag_ui_workflow_router, you can also define frontend tools. They get called by the agent
but handled on the frontend.
def hello_world(name: str) -> str:
return "called hello_world"
agentic_chat_router = get_ag_ui_workflow_router(
llm=OpenAI(model="gpt-4.1"),
frontend_tools=[hello_world]
)
You can then handle this tool with CopilotKit's useCopilotAction
// page.tsx
import { useCopilotAction } from "@copilotkit/core";
const Main = () => {
// ...
useCopilotAction({
name: "say_hello",
description: "Say hello to the name, make sure to get it first",
available: "remote",
parameters: [
{
name: "name",
type: "string",
required: true,
description: "The name to say hi to",
},
],
handler: ({ description }: { description: string }) => {
return "Hello, Tyler"; // tool_result
},
});
// ...
};
src/lib/canvas/types.tsCardType unionsrc/components/canvas/CardRenderer.tsxagent/agent/agent.pysrc/app/page.tsxFIELD_SCHEMA constantCardRenderer.tsxset[Type]Field[Number]src/app/globals.cssIf you encounter errors while setting up the project, make sure you have all the Prerequisites installed. Missing prerequisites like Node.js, Python, or uv can cause installation or build failures.
If you see "I'm having trouble connecting to my tools", make sure:
agent/.envIf you see "[Errno 48] Address already in use":
lsof -ti:9000 | xargs kill -9lsof -ti:3000 | xargs kill -9If the canvas and AI seem out of sync:
If Google Sheets sync is not working:
agent/.envCOMPOSIO_USER_ID is set (defaults to "default")If you encounter Python import errors:
cd agent
uv sync
If issues persist, recreate the virtual environment:
cd agent
rm -rf .venv
uv venv
uv sync
Feel free to submit issues and enhancement requests! This starter is designed to be easily extensible.
This project is licensed under the MIT License - see the LICENSE file for details.
[!IMPORTANT] Some features are still under active development and may not yet work as expected. If you encounter a problem using this template, please report an issue to this repository.