multimodal/omni-tars/core/README.md
This document summarizes the architecture and usage of the omni-tars core package to build a composable multi-agent system.
Mix and match capabilities as needed:
// Full-featured agent
const agent = AgentBuilder.create()
.addPlugin(mcpPlugin)
.addPlugin(guiPlugin)
.addPlugin(codePlugin)
.build();
import { AgentPlugin } from '@omni-tars/core';
export class MyCustomPlugin implements AgentPlugin {
readonly name = 'my-custom-agent';
readonly environmentSection = '<CUSTOM_ENVIRONMENT>...</CUSTOM_ENVIRONMENT>';
async initialize(): Promise<void> {
// Initialize your plugin
}
getTools(): ToolInterface[] {
// Return tools provided by this plugin
}
// Optional lifecycle hooks
onLLMRequest?(id: string, payload: any): void | Promise<void> {}
onLLMResponse?(id: string, payload: any): void | Promise<void> {}
onEachAgentLoopStart?(): void | Promise<void> {}
onAgentLoopEnd?(): void | Promise<void> {}
}
export class MyCustomToolCallEngineProvider extends ToolCallEngineProvider<MyCustomToolCallEngine> {
readonly name = 'my-custom-engine';
readonly priority = 70;
readonly description = 'My custom tool call engine for specific tasks';
protected createEngine(): MyCustomToolCallEngine {
return new MyCustomToolCallEngine();
}
canHandle(context: ToolCallEngineContext): boolean {
return context.tools.some(tool =>
tool.function.name.includes('my_special_tool')
);
}
}
import { codePlugin, CodeToolCallEngineProvider } from '@omni-tars/code-agent';
import { mcpPlugin, McpToolCallEngineProvider } from '@omni-tars/mcp-agent';
import { guiPlugin, GuiToolCallEngineProvider } from '@omni-tars/gui-agent';
import { ComposableAgent, createComposableToolCallEngineFactory } from '@omni-tars/core';
const toolCallEngine = createComposableToolCallEngineFactory({
engines: [
new GuiToolCallEngineProvider(),
new McpToolCallEngineProvider(),
new CodeToolCallEngineProvider(),
],
});
const agent = new ComposableAgent({
name: 'Omni Agent',
plugins: [mcpPlugin, guiPlugin, codePlugin],
toolCallEngine,
});
multimodal/omni-tars/
āāā core/ # Composable architecture
ā āāā src/
ā ā āāā types.ts # Plugin interfaces
ā ā āāā ComposableAgent.ts
ā ā āāā AgentComposer.ts
ā ā āāā AgentBuilder.ts
ā ā āāā environments/ # Modular environments
ā ā ā āāā code.ts
ā ā ā āāā mcp.ts
ā ā ā āāā computer.ts
ā ā āāā examples/ # Usage examples
ā ā āāā test/ # Tests
āāā mcp-agent/ # MCP plugin
ā āāā src/
ā āāā McpAgentPlugin.ts
āāā gui-agent/ # GUI plugin
ā āāā src/
ā āāā GuiAgentPlugin.ts
āāā code-agent/ # Code plugin
āāā src/
āāā CodeAgentPlugin.ts
Split multimodal/omni-tars/core/src/environments/prompt.ts into three modular sections:
CODE_ENVIRONMENT - Bash execution, file editing, Jupyter notebooksMCP_ENVIRONMENT - Search and web browsing capabilitiesCOMPUTER_USE_ENVIRONMENT - GUI interaction and screen controlinitialize() - Plugin initializationonLLMRequest() - Before each LLM requestonLLMResponse() - After each LLM responseonEachAgentLoopStart() - Start of each agent looponAgentLoopEnd() - End of each agent loopMultiple composition patterns supported: