apps/docs/content/starter-kits/agent.mdx
To build with an agent starter kit, run this command in your terminal:
npm create tldraw@latest -- --template agent
The Agent starter kit is perfect for building:
With its default configuration, the agent can perform the following actions:
To make decisions on what to do, the agent gathers information from various sources:
The agent captures both visual screenshots and structured shape data from the canvas. This dual approach gives the AI model context about what's currently displayed, including spatial relationships, visual appearance, and semantic meaning of shapes. In this way, the AI understands both the visual layout and the underlying data structure.
The agent's "eyes" are defined by PromptPartUtil classes that gather different types of context, from user messages and canvas screenshots to shape data and interaction history.
The agent performs canvas operations through a modular action system. Each action type handles specific canvas modifications, from creating simple shapes to complex multi-step operations like aligning, distributing, or arranging groups of shapes. The system includes actions for creating shapes, updating properties, manipulating selections, and coordinating multi-step workflows, each with validation, execution, error handling, and chat-panel presentation logic.
The agent's "hands" are defined by AgentActionUtil classes that specify what operations the agent can perform on the canvas.
The agent operates in one of several modes, each with its own set of capabilities tailored to a specific kind of task. The default working mode has full access to the canvas. A narrower mode might strip away drawing actions and add critique-specific ones for reviewing work, or swap in different tools for a planning phase. Modes let the same agent take on different roles at different points in a task.
AI responses stream in real-time, allowing users to see the agent's thinking process and canvas modifications as they happen. The streaming system handles partial responses gracefully and provides immediate visual feedback. Shapes get created, updated, and deleted incrementally as each action finishes streaming, so the canvas stays responsive throughout even long, multi-step tasks.
The agent's state is organized into focused managers, each responsible for a single concern: chat history, model selection, contextual shapes, the todo list, mode transitions, and more. This decomposition keeps the agent's behavior easy to reason about as the template grows, and adding a new capability means adding a new manager rather than expanding a monolith.
This starter kit supports multiple AI providers (Anthropic, OpenAI, Google) with consistent interfaces. The system abstracts provider differences, making it easy to switch models or use different providers for different types of operations. Adding support for a new provider doesn't require changes to the rest of the system.
Aside from using the chat panel UI, you can also prompt the agent programmatically.
Call the prompt() method to start an agentic loop. The agent will continue until it has finished the task you've given it.
// Inside a component wrapped by TldrawAgentAppProvider
const agent = useAgent()
agent.prompt('Draw a cat')
You can optionally specify further details about the request in the form of an AgentInput object:
agent.prompt({
message: 'Draw a cat in this area',
bounds: {
x: 0,
y: 0,
w: 300,
h: 400,
},
})
There are more methods on the TldrawAgent class that can help when building an agentic app:
agent.cancel() - Cancel the agent's current task.agent.reset() - Reset the agent's chat and memory.agent.request(input) - Send a single request to the agent and handle its response without entering into an agentic loop.The agent's behavior is defined in client/modes/AgentModeDefinitions.ts. The AGENT_MODE_DEFINITIONS array contains mode definitions. Each mode has two arrays:
parts determine what the agent can see.actions determine what the agent can do.Add, edit or remove an entry in either array to change what the agent can see or do in a given mode.
Modes control which parts and actions the agent has access to at any given time. They're defined in client/modes/AgentModeDefinitions.ts.
The default working mode includes all standard capabilities. You can create additional modes with different subsets of parts and actions.
Call agent.mode.setMode(modeType) to change modes during a prompt. To control each mode's lifecycle, implement any of the lifecycle hooks in client/modes/AgentModeChart.ts:
onEnter(agent, fromMode) - runs when the agent enters a mode.onExit(agent, toMode) - runs when the agent exits a mode.onPromptStart(agent, request) - runs when a prompt commences.onPromptEnd(agent, request) - runs when a prompt ends.onPromptCancel(agent, request) - runs when a prompt is canceled.Change what the agent can see by adding, editing or removing a prompt part.
Prompt parts assemble the prompt that the model receives, with each part adding a different piece of information. This includes the user's message, the model name, the system prompt, chat history and more.
A prompt part has two pieces: a PromptPartUtil class on the client that gathers data, and a PromptPartDefinition on the worker that turns that data into messages.
This example shows how to let the model see what the current time is.
First, define a prompt part type:
interface TimePart extends BasePromptPart<'time'> {
time: string
}
Then, create a prompt part util in client/parts/:
export const TimePartUtil = registerPromptPartUtil(
class TimePartUtil extends PromptPartUtil<TimePart> {
static override type = 'time' as const
override getPart(): TimePart {
return {
type: 'time',
time: new Date().toLocaleTimeString(),
}
}
}
)
Next, create the prompt part definition in shared/schema/PromptPartDefinitions.ts:
export const TimePartDefinition: PromptPartDefinition<TimePart> = {
type: 'time',
priority: -100,
buildContent({ time }: TimePart) {
return [`The user's current time is: ${time}`]
},
}
To enable the prompt part, import its util in client/modes/AgentModeDefinitions.ts and add TimePartUtil.type to a mode's parts array. Its methods will be used to assemble its data and send it to the model.
getPart() - Gather any data needed to construct the prompt.buildContent() - Turn the data into messages to send to the model.There are other fields available on the PromptPartDefinition interface that you can override for more granular control.
priority - Control where this prompt part will appear in the list of messages sent to the model. Higher priority appears later in the prompt.getModelName() - Determine which AI model to use.buildMessages() - Manually override how prompt messages are constructed from the prompt part.Change what the agent can do by adding, editing or removing an AgentActionUtil.
Agent action utils define which actions the agent can perform. Each AgentActionUtil adds a different capability.
This example shows how to allow the agent to clear the screen.
First, define an agent action by creating a schema for it in shared/schema/AgentActionSchemas.ts:
export const ClearAction = z
// All agent actions must have a _type field
// The underscore encourages the model to put this field first
.object({
_type: z.literal('clear'),
})
// A title and description tell the model what the action does
.meta({
title: 'Clear',
description: 'The agent deletes all shapes on the canvas.',
})
// Infer the action's type
export type ClearAction = z.infer<typeof ClearAction>
Then, create an agent action util in client/actions/:
export const ClearActionUtil = registerActionUtil(
class ClearActionUtil extends AgentActionUtil<ClearAction> {
static override type = 'clear' as const
override applyAction(action: Streaming<ClearAction>) {
// Don't do anything until the action has finished streaming
if (!action.complete) return
// Get the editor
const { editor } = this
// Delete all shapes on the page
const shapes = editor.getCurrentPageShapes()
editor.deleteShapes(shapes)
}
}
)
To enable the agent action, import its util in client/modes/AgentModeDefinitions.ts and add ClearActionUtil.type to a mode's actions array. Its method will be used to execute the action.
applyAction() - Execute the action.There are other methods available on the AgentActionUtil class that you can override for more granular control.
getInfo() - Determine how the action gets displayed in the chat panel UI.savesToHistory() - Control whether actions get saved to chat history or not.sanitizeAction() - Apply transformations to the action before saving it to history and applying it. More details on transformations below.Configure the icon and description of an action in the chat panel UI using the getInfo() method.
override getInfo() {
return {
icon: 'trash' as const,
description: 'Cleared the canvas',
}
}
You can make an action collapsible by adding a summary property.
override getInfo() {
return {
summary: 'Cleared the canvas',
description: 'After much consideration, the agent decided to clear the canvas',
}
}
To customize an action's appearance via CSS, define styles for the agent-action-type-{TYPE} class where {TYPE} is the type of the action.
.agent-action-type-clear {
color: red;
}
Managers are classes that extend TldrawAgent or TldrawAgentApp with a focused responsibility, such as chat history, model selection, or context management. They're available as properties on the agent instance, for example agent.chat, agent.modelName, and agent.context.
To add a custom manager, extend BaseAgentManager or BaseAgentAppManager and attach it to the agent in client/agent/TldrawAgent.ts.
Utils use a self-registration pattern. When you create a new PromptPartUtil or AgentActionUtil, wrap it with a registration function:
export const MyPartUtil = registerPromptPartUtil(
class MyPartUtil extends PromptPartUtil<MyPart> {
// ...
}
)
This ensures the util is discovered automatically when its module is imported in AgentModeDefinitions.ts.
Different modes can implement actions with the same _type. For example, a team-member mode and a solo mode might both have a mark-task-done action, but with different implementations.
To register a mode-specific action util, pass the forModes option:
export const MarkTeamMemberTaskDoneActionUtil = registerActionUtil(
class MarkTeamMemberTaskDoneActionUtil extends AgentActionUtil<MarkTeamMemberTaskDoneAction> {
static override type = 'mark-task-done' as const
override applyAction(action: Streaming<MarkTeamMemberTaskDoneAction>) {
// Team member-specific implementation
}
},
{ forModes: ['team-member'] }
)
Default schemas are auto-registered when exported from AgentActionSchemas.ts. For mode-specific schemas, call registerActionSchema explicitly with the forModes option.
See the template README for a full worked example.
You can let the agent work over multiple turns by scheduling further work using the schedule method as part of an action.
This example shows how to schedule an extra step for adding detail to the canvas.
override applyAction(action: Streaming<AddDetailAction>) {
if (!action.complete) return
if (!this.agent) return
this.agent.schedule('Add more detail to the canvas.')
}
As with the prompt method, you can specify further details about the request.
agent.schedule({
message: 'Add more detail in this area.',
bounds: { x: 0, y: 0, w: 100, h: 100 },
})
You can schedule multiple things by calling the schedule() method more than once.
agent.schedule('Add more detail to the canvas.')
agent.schedule('Check for spelling mistakes.')
To interrupt the agent with a new prompt instead of waiting for the current prompt to end, use the interrupt method. interrupt also lets you specify a mode to transition into.
override applyAction(action: Streaming<EnterReviewingModeAction>) {
if (!action.complete) return
this.agent.interrupt({
mode: 'reviewing',
input: {
message: 'Review the new area thoroughly for any mistakes',
bounds: action.bounds,
},
})
}
To let the agent retrieve information from an external API, fetch the data within applyAction and schedule a follow-up request with any data you want the agent to have access to.
override async applyAction(action: Streaming<CountryInfoAction>) {
if (!action.complete) return
if (!this.agent) return
// Fetch from the external API
const data = await fetchCountryInfo(action.code)
// Schedule a follow-up request with the data
this.agent.schedule({ data: [data] })
}
The model can make mistakes. Sometimes this is due to hallucinations, and sometimes this is due to the canvas changing since the last time the model saw it. Either way, an incoming action might contain invalid data by the time you receive it.
To correct incoming mistakes, apply fixes in the sanitizeAction() method of an action util. They'll get carried out before the action is applied to the editor or saved to chat history.
For example, ensure that a shape ID received from the model refers to an existing shape by using the ensureShapeIdExists() method.
override sanitizeAction(action: Streaming<DeleteAction>, helpers: AgentHelpers) {
if (!action.complete) return action
// Ensure the shape ID refers to an existing shape
action.shapeId = helpers.ensureShapeIdExists(action.shapeId)
// If the shape ID doesn't refer to an existing shape, cancel the action
if (!action.shapeId) return null
return action
}
The AgentHelpers class contains more helpers for sanitizing data received from the model.
ensureShapeIdExists() - Ensure that a shape ID refers to a real shape. Useful for interacting with existing shapes.ensureShapeIdIsUnique() - Ensure that a shape ID is unique. Useful for creating new shapes.ensureValueIsVec(), ensureValueIsNumber() - Ensure that a value is a certain type. Useful for more complex actions where the model is more likely to make mistakes.By default, every position sent to the model is offset by the starting position of the current chat.
To apply this offset to a position sent to the model, use the applyOffsetToVec method.
override getPart(request: AgentRequest, helpers: AgentHelpers): ViewportCenterPart {
if (!this.editor) return { part: 'user-viewport-center', center: null }
// Get the center of the user's viewport
const viewportCenter = this.editor.getViewportBounds().center
// Apply the chat's offset to the vector
const offsetViewportCenter = helpers.applyOffsetToVec(viewportCenter)
// Return the prompt part
return {
part: 'user-viewport-center',
center: offsetViewportCenter,
}
}
To remove the offset from a position received from the model, use the removeOffsetFromVec method.
override applyAction(action: Streaming<MoveAction>, helpers: AgentHelpers) {
if (!action.complete) return
// Remove the offset from the position
const position = helpers.removeOffsetFromVec({ x: action.x, y: action.y })
// Do something with the position...
}
It's a good idea to round numbers before sending them to the model. If you want to be able to restore the original number later, use the roundAndSaveNumber and unroundAndRestoreNumber methods.
// In `getPart`...
const roundedX = helpers.roundAndSaveNumber(x, 'my_key_x')
const roundedY = helpers.roundAndSaveNumber(y, 'my_key_y')
// In `applyAction`...
const unroundedX = helpers.unroundAndRestoreNumber(x, 'my_key_x')
const unroundedY = helpers.unroundAndRestoreNumber(y, 'my_key_y')
To round all the numbers on a shape, use the roundShape and unroundShape methods. See the Send shapes to the model section below for more details.
By default, the agent converts tldraw shapes to various simplified formats to improve the model's understanding and performance.
There are three main formats used in this starter:
BlurryShape - The format for shapes within the agent's viewport. It contains a shape's bounds, its ID, its type, and any text it contains. The "blurry" name refers to the fact that the agent can't make out the details of shapes from this format. Instead, it gives the model an overview of what it's looking at.FocusedShape - The format for shapes that the agent is focusing on, such as shapes you've manually added to its context. The format contains most of a shape's properties, including color, fill, alignment, and any other shape-specific information. This is also the format the model outputs when creating shapes.PeripheralShapeCluster - The format for shapes outside the agent's viewport. Nearby shapes are grouped together into clusters, each with the group's bounds and a count of how many shapes are inside it. This is the least detailed format. Its role is to give the model an awareness of shapes that elsewhere on the page.To send the model some shapes in one of these formats, use one of the conversion functions found within the format folder, such as convertTldrawShapeToFocusedShape().
This example picks one random shape on the canvas and sends it to the model in the Focused format.
override getPart(request: AgentRequest, helpers: AgentHelpers): RandomShapePart {
const { editor } = this
if (!this.editor) return { type: 'random-shape', shape: null}
// Get a random shape
const shapes = editor.getCurrentPageShapes()
const randomShape = shapes[Math.floor(Math.random() * shapes.length)]
// Convert the shape to the Focused format
const focusedShape = convertTldrawShapeToFocusedShape(randomShape, editor)
// Normalize the shape's position
const offsetShape = helpers.applyOffsetToShape(focusedShape)
const roundedShape = helpers.roundShape(offsetShape)
return {
type: 'random-shape',
shape: roundedShape,
}
}
To change the default system prompt, edit the sections in worker/prompt/sections/. These sections are assembled by worker/prompt/buildSystemPrompt.ts.
The system prompt is rebuilt for each step in the agentic loop depending on which actions and parts are available in the agent's current mode. To give the model more detailed instructions for how to use any custom actions or parts you add, edit worker/prompt/sections/rules-section.ts.
You can set an agent's model by calling setModelName on the modelName manager.
agent.modelName.setModelName('gemini-3-flash-preview')
To override an agent's model, specify a different model name with a request.
agent.prompt({
modelName: 'gemini-3-flash-preview',
message: 'Draw a diagram of a volcano.',
})
You can conditionally override the model name by overriding the getModelName() method on any PromptPartDefinition.
override getModelName(part: MyCustomPromptPart) {
return part.fastMode ? 'gemini-3-flash-preview' : 'claude-sonnet-4-5'
}
To add support for a different model, add the model's definition to AGENT_MODEL_DEFINITIONS in the shared/models.ts file.
'claude-sonnet-4-5': {
name: 'claude-sonnet-4-5',
id: 'claude-sonnet-4-5',
provider: 'anthropic',
}
If you need to add any extra setup or configuration for your provider, you can add it to the worker/do/AgentService.ts file.
If your app includes custom shapes, the agent will be able to see, move, delete, resize, rotate and arrange them with no extra setup. However, you might want to also let the agent create and edit them, and read their custom properties.
To support custom shapes, you have two main options:
Add an action that lets the agent create your custom shape.
See the Let the agent create custom shapes with an action section below.
Add your custom shape to the schema so that the agent read, edit and create it like any other shape.
See the Add your custom shape to the schema section below.
To add partial support for a custom shape, let the agent create it with an agent action. For example, this action lets the agent create a custom "sticker" shape:
// In shared/schema/AgentActionSchemas.ts
export const StickerAction = z
.object({
_type: z.literal('sticker'),
stickerType: z.enum(['❤️', '⭐']),
x: z.number(),
y: z.number(),
})
.meta({
title: 'Sticker',
description: 'Add a sticker to the canvas.',
})
export type StickerAction = z.infer<typeof StickerAction>
Define how the action gets applied to the canvas by creating an action util:
// In client/actions/StickerActionUtil.ts
export const StickerActionUtil = registerActionUtil(
class StickerActionUtil extends AgentActionUtil<StickerAction> {
static override type = 'sticker' as const
// Tell the model how to display the action in chat history
override getInfo(action: Streaming<StickerAction>) {
return {
icon: 'pencil' as const,
description: 'Added a sticker',
}
}
// Execute the action
override applyAction(action: Streaming<StickerAction>, helpers: AgentHelpers) {
if (!action.complete) return
if (!this.editor) return
// Normalize the position
const position = helpers.removeOffsetFromVec({ x: action.x, y: action.y })
// Create the custom shape
this.editor.createShape({
type: 'sticker',
id: createShapeId(),
x: position.x,
y: position.y,
props: { stickerType: action.stickerType },
})
}
}
)
To let the agent see the custom properties of your custom shape, add it to the schema in shared/format/FocusedShape.ts.
For example, here's a schema for a custom sticker shape.
const FocusedStickerShape = z
.object({
// Required properties
_type: z.literal('sticker'),
note: z.string(),
shapeId: z.string(),
// Custom properties
stickerType: z.enum(['❤️', '⭐']),
x: z.number(),
y: z.number(),
})
.meta({
// Information about the shape to give to the agent
title: 'Sticker Shape',
description:
'A sticker shape is a small symbol stamped onto the canvas. There are two types of stickers: heart and star.',
})
The _type and shapeId properties are required so that the app can identify your shape. The note property is also required. The agent uses it to leave notes for itself.
For optional properties, it's worth considering how the agent should see your custom shape. You might want to leave out some properties and focus on showing the most important ones. It's also best to keep them in alphabetical order for better performance with Gemini models.
Enable your custom shape schema by adding it to the list of FOCUSED_SHAPES in the same file.
const FOCUSED_SHAPES = [
FocusedDrawShape,
FocusedGeoShape,
FocusedLineShape,
FocusedTextShape,
FocusedArrowShape,
FocusedNoteShape,
FocusedUnknownShape,
// Our custom shape
FocusedStickerShape,
] as const
Tell the app how to convert your custom shape into the FocusedShape format by adding a case in shared/format/convertTldrawShapeToFocusedShape.ts:
export function convertTldrawShapeToFocusedShape(editor: Editor, shape: TLShape): FocusedShape {
switch (shape.type) {
// ...
case 'sticker':
const bounds = getShapeBounds(shape)
return {
_type: 'sticker',
note: (shape.meta.note as string) ?? '',
shapeId: convertTldrawIdToSimpleId(shape.id),
stickerType: shape.props.stickerType,
x: bounds.x,
y: bounds.y,
}
// ...
}
}
To allow the agent to edit your custom shape's properties, tell the app how to convert your shape from the FocusedShape format that the model outputs to the actual format of your shape:
export function convertFocusedShapeToTldrawShape(
editor: Editor,
focusedShape: FocusedShape,
{ defaultShape }: { defaultShape: Partial<TLShape> }
) {
switch (focusedShape._type) {
// ...
case 'sticker':
const shapeId = convertSimpleIdToTldrawId(focusedShape.shapeId)
return {
shape: {
id: shapeId,
x: focusedShape.x,
y: focusedShape.y,
// ...
props: {
// ...
stickerType: focusedShape.stickerType,
},
meta: {
note: focusedShape.note ?? '',
},
},
}
// ...
}
}
Multiplayer Starter Kit: Use a tldraw multiplayer sync starter kit to build multi-user agent environments.
Shape Utilities: Learn how to create custom shapes and extend tldraw's shape system with advanced geometry, rendering, and interaction patterns.
Editor State Management: Learn how to work with tldraw's reactive state system, editor lifecycle, and event handling for complex canvas applications.
If you build something great, please share it with us in our #show-and-tell channel on Discord. We want to see what you've built!