docs/source/guides/orchestration.rst
.. _agent_routing:
Building multi-agent systems allow you to route requests across multiple specialized agents, each designed to handle specific types of tasks. Plano makes it easy to build and scale these systems by managing the orchestration layer—deciding which agent(s) should handle each request—while you focus on implementing individual agent logic.
This guide shows you how to configure and implement multi-agent orchestration in Plano using a real-world example: a Travel Booking Assistant that routes queries to specialized agents for weather and flights.
Plano's orchestration layer analyzes incoming prompts and routes them to the most appropriate agent based on user intent and conversation context. The workflow is:
Plano-Orchestrator-30B-A3B <https://huggingface.co/collections/katanemo/plano-orchestrator>_, which offers performance of foundation models at 1/10th the cost. The LLM routes the request to the most suitable agent configured in your system—such as a weather agent or flight agent.inner loop <agents> until the task is complete. This means the agent autonomously calls models, invokes tools, processes data, and reasons about next steps—all within its specialized domain—before returning the final response.Let's walk through a complete multi-agent system: a Travel Booking Assistant that helps users plan trips by providing weather forecasts and flight information. This system uses two specialized agents:
Configure your agents in the listeners section of your plano_config.yaml:
.. literalinclude:: ../resources/includes/agents/agents_config.yaml :language: yaml :linenos: :caption: Travel Booking Multi-Agent Configuration
Key Configuration Elements:
type: agent tells Plano to perform intent analysis and routing for incoming requests.id, description (used for routing decisions)plano_orchestrator_v1 router uses Plano-Orchestrator to analyze user intent and select the appropriate agent.filter chains <filter_chain> to agents for guardrails, query rewriting, or context enrichment.Writing Effective Agent Descriptions
Agent descriptions are critical—they're used by Plano-Orchestrator to make routing decisions. Effective descriptions should include:
Clear introduction: A concise statement explaining what the agent is and its primary purpose
Capabilities section: A bulleted list of specific capabilities, including:
Here's an example of a well-structured agent description:
.. code-block:: yaml
- id: weather_agent
description: |
WeatherAgent is a specialized AI assistant for real-time weather information
and forecasts. It provides accurate weather data for any city worldwide using
the Open-Meteo API, helping travelers plan their trips with up-to-date weather
conditions.
Capabilities:
* Get real-time weather conditions and multi-day forecasts for any city worldwide
* Provides current temperature, weather conditions, sunrise/sunset times
* Provides detailed weather information including multi-day forecasts
* Understands conversation context to resolve location references from previous messages
* Handles weather-related questions including "What's the weather in [city]?"
* When queries include both weather and other travel questions (e.g., flights),
this agent answers ONLY the weather part
.. note::
We will soon support "Agents as Tools" via Model Context Protocol (MCP), enabling agents to dynamically discover and invoke other agents as tools. Track progress on GitHub Issue #646 <https://github.com/katanemo/archgw/issues/646>_.
Agents are HTTP services that receive routed requests from Plano. Each agent implements the OpenAI Chat Completions API format, making them compatible with standard LLM clients.
Agent Structure ^^^^^^^^^^^^^^^
Let's examine the Weather Agent implementation:
.. literalinclude:: ../resources/includes/agents/weather.py :language: python :linenos: :lines: 262-283 :caption: Weather Agent - Core Structure
Key Points:
/v1/chat/completions endpoint that matches OpenAI's API formatLLM_GATEWAY_ENDPOINT) for all LLM callsrequest_body.messagesInformation Extraction with LLMs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Agents use LLMs to extract structured information from natural language queries. This enables them to understand user intent and extract parameters needed for API calls.
The Weather Agent extracts location information:
.. literalinclude:: ../resources/includes/agents/weather.py :language: python :linenos: :lines: 73-119 :caption: Weather Agent - Location Extraction
The Flight Agent extracts more complex information—origin, destination, and dates:
.. literalinclude:: ../resources/includes/agents/flights.py :language: python :linenos: :lines: 69-120 :caption: Flight Agent - Flight Information Extraction
Key Points:
gpt-4o-mini) for extraction tasksCalling External APIs ^^^^^^^^^^^^^^^^^^^^^^
After extracting information, agents call external APIs to fetch real-time data:
.. literalinclude:: ../resources/includes/agents/weather.py :language: python :linenos: :lines: 136-197 :caption: Weather Agent - External API Call
The Flight Agent calls FlightAware's AeroAPI:
.. literalinclude:: ../resources/includes/agents/flights.py :language: python :linenos: :lines: 156-260 :caption: Flight Agent - External API Call
Key Points:
httpx.AsyncClient) for non-blocking API callsPreparing Context and Generating Responses ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Agents combine extracted information, API data, and conversation history to generate responses:
.. literalinclude:: ../resources/includes/agents/weather.py :language: python :linenos: :lines: 290-370 :caption: Weather Agent - Context Preparation and Response Generation
Key Points:
Write Clear Agent Descriptions
Agent descriptions are used by Plano-Orchestrator to make routing decisions. Be specific about what each agent handles:
.. code-block:: yaml
# Good - specific and actionable
- id: flight_agent
description: Get live flight information between airports using FlightAware AeroAPI. Shows real-time flight status, scheduled/estimated/actual departure and arrival times, gate and terminal information, delays, aircraft type, and flight status. Automatically resolves city names to airport codes (IATA/ICAO). Understands conversation context to infer origin/destination from follow-up questions.
# Less ideal - too vague
- id: flight_agent
description: Handles flight queries
Use Conversation Context Effectively
Include conversation history in your extraction and response generation:
.. code-block:: python
# Include conversation context for extraction
conversation_context = []
for msg in messages:
conversation_context.append({"role": msg.role, "content": msg.content})
# Use recent context (last 10 messages)
context_messages = conversation_context[-10:] if len(conversation_context) > 10 else conversation_context
Route LLM Calls Through Plano's Model Proxy
Always route LLM calls through Plano's :ref:Model Proxy <llm_providers> for consistent responses, smart routing, and rich observability:
.. code-block:: python
openai_client_via_plano = AsyncOpenAI(
base_url=LLM_GATEWAY_ENDPOINT, # Plano's LLM gateway
api_key="EMPTY",
)
response = await openai_client_via_plano.chat.completions.create(
model="openai/gpt-4o",
messages=messages,
stream=True,
)
Handle Errors Gracefully
Provide fallback values and clear error messages:
.. code-block:: python
async def get_weather_data(request: Request, messages: list, days: int = 1):
try:
# ... extraction and API logic ...
location = response.choices[0].message.content.strip().strip("\"'`.,!?")
if not location or location.upper() == "NOT_FOUND":
location = "New York" # Fallback to default
return weather_data
except Exception as e:
logger.error(f"Error getting weather data: {e}")
return {"location": "New York", "weather": {"error": "Could not retrieve weather data"}}
Use Appropriate Models for Tasks
Use smaller, faster models for extraction tasks and larger models for final responses:
.. code-block:: python
# Extraction: Use smaller, faster model
LOCATION_MODEL = "openai/gpt-4o-mini"
# Final response: Use larger, more capable model
WEATHER_MODEL = "openai/gpt-4o"
Stream Responses
Stream responses for better user experience:
.. code-block:: python
async def invoke_weather_agent(request: Request, request_body: dict, traceparent_header: str = None):
# ... prepare messages with weather data ...
stream = await openai_client_via_plano.chat.completions.create(
model=WEATHER_MODEL,
messages=response_messages,
temperature=request_body.get("temperature", 0.7),
max_tokens=request_body.get("max_tokens", 1000),
stream=True,
extra_headers=extra_headers,
)
async for chunk in stream:
if chunk.choices:
yield f"data: {chunk.model_dump_json()}\n\n"
yield "data: [DONE]\n\n"
Multi-agent orchestration is particularly powerful for:
Travel and Booking Systems
Route queries to specialized agents for weather and flights:
.. code-block:: yaml
agents:
- id: weather_agent
description: Get real-time weather conditions and forecasts
- id: flight_agent
description: Search for flights and provide flight status
Customer Support
Route common queries to automated support agents while escalating complex issues:
.. code-block:: yaml
agents:
- id: tier1_support
description: Handles common FAQs, password resets, and basic troubleshooting
- id: tier2_support
description: Handles complex technical issues requiring deep product knowledge
- id: human_escalation
description: Escalates sensitive issues or unresolved problems to human agents
Sales and Marketing
Direct leads and inquiries to specialized sales agents:
.. code-block:: yaml
agents:
- id: product_recommendation
description: Recommends products based on user needs and preferences
- id: pricing_agent
description: Provides pricing information and quotes
- id: sales_closer
description: Handles final negotiations and closes deals
Technical Documentation and Support
Combine RAG agents for documentation lookup with specialized troubleshooting agents:
.. code-block:: yaml
agents:
- id: docs_agent
description: Retrieves relevant documentation and guides
filter_chain:
- query_rewriter
- context_builder
- id: troubleshoot_agent
description: Diagnoses and resolves technical issues step by step
By default, Plano uses a hosted Plano-Orchestrator endpoint. To self-host the orchestrator model, you can serve it using vLLM on a server with an NVIDIA GPU.
.. note:: vLLM requires a Linux server with an NVIDIA GPU (CUDA). For local development on macOS, a GGUF version for Ollama is coming soon.
The following model variants are available on HuggingFace:
Plano-Orchestrator-4B <https://huggingface.co/katanemo/Plano-Orchestrator-4B>_ — lighter model, suitable for development and testingPlano-Orchestrator-4B-FP8 <https://huggingface.co/katanemo/Plano-Orchestrator-4B-FP8>_ — FP8 quantized 4B model, lower memory usagePlano-Orchestrator-30B-A3B <https://huggingface.co/katanemo/Plano-Orchestrator-30B-A3B>_ — full-size model for productionPlano-Orchestrator-30B-A3B-FP8 <https://huggingface.co/katanemo/Plano-Orchestrator-30B-A3B-FP8>_ — FP8 quantized 30B model, recommended for production deploymentsUsing vLLM
1. **Install vLLM**
.. code-block:: bash
pip install vllm
2. **Download the model and chat template**
.. code-block:: bash
pip install huggingface_hub
huggingface-cli download katanemo/Plano-Orchestrator-4B
3. **Start the vLLM server**
For the 4B model (development):
.. code-block:: bash
vllm serve katanemo/Plano-Orchestrator-4B \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.3 \
--tokenizer katanemo/Plano-Orchestrator-4B \
--chat-template chat_template.jinja \
--served-model-name katanemo/Plano-Orchestrator-4B \
--enable-prefix-caching
For the 30B-A3B-FP8 model (production):
.. code-block:: bash
vllm serve katanemo/Plano-Orchestrator-30B-A3B-FP8 \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.9 \
--tokenizer katanemo/Plano-Orchestrator-30B-A3B-FP8 \
--chat-template chat_template.jinja \
--max-model-len 32768 \
--served-model-name katanemo/Plano-Orchestrator-30B-A3B-FP8 \
--enable-prefix-caching
4. **Configure Plano to use the local orchestrator**
Use the model name matching your ``--served-model-name``:
.. code-block:: yaml
overrides:
agent_orchestration_model: plano/katanemo/Plano-Orchestrator-4B
model_providers:
- model: katanemo/Plano-Orchestrator-4B
provider_interface: plano
base_url: http://<your-server-ip>:8000
5. **Verify the server is running**
.. code-block:: bash
curl http://localhost:8000/health
curl http://localhost:8000/v1/models
Next Steps
----------
* Learn more about :ref:`agents <agents>` and the inner vs. outer loop model
* Explore :ref:`filter chains <filter_chain>` for adding guardrails and context enrichment
* See :ref:`observability <observability>` for monitoring multi-agent workflows
* Review the :ref:`LLM Providers <llm_providers>` guide for model routing within agents
* Check out the complete `Travel Booking demo <https://github.com/katanemo/plano/tree/main/demos/agent_orchestration/travel_agents>`_ on GitHub
.. note::
To observe traffic to and from agents, please read more about :ref:`observability <observability>` in Plano.
By carefully configuring and managing your Agent routing and hand off, you can significantly improve your application's responsiveness, performance, and overall user satisfaction.