Chat System Architecture

flowchart TD
    subgraph Client["📱 Flutter App"]
        Q[User Question]
    end

    subgraph Backend["🖥️ FastAPI Backend"]
        Router{LangGraph
Router}
    end

    Q --> Router

    Router -->|Simple| NC[No Context Path]
    Router -->|Context Needed| A[Agentic Path]
    Router -->|Persona App| P[Persona Path]

    NC --> LLM1[Direct LLM
Response]
    A --> Tools[Tool Calls
22+ tools]
    Tools --> LLM2[LLM with
Context]
    P --> LLM3[Persona LLM
Response]

    LLM1 --> Stream[Streaming Response]
    LLM2 --> Stream
    LLM3 --> Stream

    Stream --> |with citations| Q

**When triggered:** Simple greetings, general advice, brainstorming questions

**Classification criteria** (from `requires_context()` function):
- Greetings: "Hi", "Hello", "How are you?"
- General knowledge: "What's the capital of France?"
- Advice without personal context: "Tips for productivity"

**Processing:**
```python
# Location: backend/utils/retrieval/graph.py
def no_context_conversation(state: ChatState, config: RunnableConfig):
    # Direct LLM call without tool access
    # Fast response, no memory retrieval
```

<Tip>
This path provides the fastest responses since no external data retrieval is needed.
</Tip>

**When triggered:** Questions requiring personal data, temporal queries, integration lookups

**Classification criteria:**
- References to "my", "I", personal data
- Temporal references: "yesterday", "last week", "this morning"
- Questions about conversations, memories, calendar, health
- Requests involving connected services

**Processing:**
```python
# Location: backend/utils/retrieval/agentic.py
# LangGraph ReAct agent with full tool access
# LLM autonomously decides which tools to call
# Can make multiple tool calls to gather comprehensive context
```

<Info>
This is the most powerful path - the LLM can call 22+ tools to gather comprehensive context before answering.
</Info>

**When triggered:** Questions directed at persona-based apps (e.g., "Ask Einstein")

**Processing:**
- Uses the app's configured `persona_prompt`
- Character-consistent responses
- May have limited tool access based on app configuration

<Note>
Persona apps can customize which tools are available, allowing for focused conversational experiences.
</Note>

# Location: backend/utils/retrieval/graph.py
def requires_context(messages: list) -> bool:
    """
    Uses GPT-4-mini for fast classification.

    Returns True if question needs:
    - Personal memories/conversations
    - Calendar/email/health data
    - Temporal context
    - User-specific information
    """

| Tool | Purpose | Key Parameters |
|------|---------|----------------|
| `get_conversations_tool` | Retrieve by date range | `start_date`, `end_date`, `limit`, `include_transcript` |
| `search_conversations_tool` | Semantic search | `query`, `start_date`, `end_date`, `limit` |
| `get_memories_tool` | Personal facts about user | `limit`, `offset` |

| Tool | Purpose |
|------|---------|
| `get_action_items_tool` | Retrieve pending tasks |
| `create_action_item_tool` | Create new task |
| `update_action_item_tool` | Mark complete/update |

| Tool | Purpose |
|------|---------|
| `get_calendar_events_tool` | Fetch events by date/person |
| `create_calendar_event_tool` | Create meetings with attendees |
| `update_calendar_event_tool` | Modify existing events |
| `delete_calendar_event_tool` | Cancel meetings |

| Tool | Service | Purpose |
|------|---------|---------|
| `get_gmail_messages_tool` | Gmail | Search emails |
| `get_whoop_sleep_tool` | Whoop | Sleep data |
| `get_whoop_recovery_tool` | Whoop | Recovery scores |
| `get_whoop_workout_tool` | Whoop | Workout history |
| `search_notion_pages_tool` | Notion | Search workspace |
| `get_twitter_tweets_tool` | Twitter/X | Recent tweets |
| `get_github_pull_requests_tool` | GitHub | Open PRs |
| `get_github_issues_tool` | GitHub | Open issues |
| `perplexity_web_search_tool` | Perplexity | Web search |

```python
# Location: backend/utils/retrieval/tools/app_tools.py
def load_app_tools(uid: str) -> List[Callable]:
    """
    Loads tools from user's enabled apps.
    Each app can define chat_tools in its configuration.
    """
```

<Tip>
See [Chat Tools for Apps](/doc/developer/apps/ChatTools) to learn how to build custom tools.
</Tip>

# Maximum 10 tool calls per question (prevents runaway loops)
# Maximum 500K tokens in context (prevents context overflow)
# 30-second timeout per external API call

sequenceDiagram
    participant Agent as 🤖 LLM Agent
    participant Tool as 🔧 Vector Search Tool
    participant Embed as 📊 OpenAI Embeddings
    participant Pine as 🌲 Pinecone
    participant Fire as 🔥 Firestore

    Agent->>Tool: search_conversations_tool(query, dates)
    Tool->>Embed: embed_query("John project discussion")
    Embed-->>Tool: [0.012, -0.034, 0.056, ...] (3,072 dims)
    Tool->>Pine: query(vector, uid filter, date range)
    Pine-->>Tool: [conv_id_456, conv_id_789] ranked by similarity
    Tool->>Fire: get_conversations_by_id(ids)
    Fire-->>Tool: Full conversation data
    Tool-->>Agent: Formatted context with citations

# Location: backend/utils/conversations/process_conversation.py
# Triggered after conversation processing completes

def save_structured_vector(uid: str, conversation: Conversation):
    """
    1. Generate embedding from conversation.structured
       (title + overview + action_items + events)
    2. Extract metadata via LLM (people, topics, entities, dates)
    3. Upsert to Pinecone with metadata
    """

# Location: backend/database/vector_db.py

def query_vectors(query: str, uid: str, starts_at: int, ends_at: int, k: int):
    """
    1. Embed query using text-embedding-3-large
    2. Query Pinecone with uid filter and optional date range
    3. Return top-k conversation IDs ranked by similarity
    """

def query_vectors_by_metadata(uid, vector, dates_filter, people, topics, entities, dates, limit):
    """
    Advanced query with metadata filters.
    Includes fallback: if no results with filters, retries without them.
    """

# From backend/utils/llm/chat.py

# Maximum 15 words per memory
# Must pass "shareability test" - worth telling someone
# Max 2 interesting + 2 system memories per conversation
# NO duplicate/near-duplicate facts
# NO mundane details (eating, sleeping, commuting)

# Tool: get_memories_tool
# Returns formatted list of known facts about user
# Used when questions ask "What do you know about me?"

# Location: backend/database/chat.py

# Chat sessions group related messages
# Each session tracks:
# - message_ids: List of message IDs
# - file_ids: Uploaded files for this session
# - openai_thread_id: For file-based chat

# Last 10 messages included in context
# Enables follow-up questions without re-stating context
# Older messages summarized or excluded

# Citation rules:
# - No space before citation: "discussed this[1]" not "discussed this [1]"
# - Citations map to conversation IDs
# - Post-processing extracts citations → memories_id field
# - Frontend displays linked conversation cards

# Location: backend/utils/llm/chat.py - _get_agentic_qa_prompt()

# 1. Current datetime in user's timezone
# 2. Tool usage instructions
# 3. DateTime formatting rules for tool calls
# 4. Conversation retrieval strategies (5-step strategy)
# 5. Citation format instructions
# 6. Memory extraction guidelines

# Good: "2024-01-19T00:00:00-08:00"
# Bad: "yesterday", "last week" (must be converted)

# The system prompt instructs the LLM to convert relative
# references to absolute ISO timestamps before tool calls

think: Searching conversations        # Tool call indicator
data: Yesterday you discussed...      # Response text chunks
done: {base64 encoded JSON}           # Final message with metadata

Route to: `agentic_context_dependent_conversation`

Agent thinks: *"Need conversations from yesterday about project with John"*

Agent calls: `search_conversations_tool`
- `query`: "John project discussion"
- `start_date`: "2024-01-19T00:00:00-08:00"
- `end_date`: "2024-01-19T23:59:59-08:00"

*"Yesterday you discussed the Q1 roadmap with John[1]. He mentioned the frontend refactoring is ahead of schedule[1][2]..."*