Session Lifecycle

Audience: Gateway developers and maintainers Source files: gateway/session.py (~1200 lines + session_*.py siblings), gateway/run.py (~5500 lines facade + run_*.py phases), gateway/config.py Last updated: 2026-06-16

Overview

A session represents a continuous conversation between the agent and one or more users on a messaging platform. The session lifecycle governs when conversations persist, when they reset, how they survive gateway restarts, and how messages queue during concurrent operations.

The session system lives primarily in two modules:

• gateway/session.py — Data model (SessionSource, SessionEntry, SessionContext), key generation (build_session_key), and the main store (SessionStore).
• gateway/run.py — Gateway runner (GatewayRunner) facade that wires sessions into the message processing pipeline; the phases live in run_*.py siblings: session housekeeping (run_watchers.py), agent caching (run_agent_cache.py), restart recovery (session_recovery.py), and message queuing (run_busy.py).

1. SessionSource — Message Origin Descriptor

SessionSource is a frozen record of where a message came from. It is attached to every incoming MessageEvent and used for routing, isolation, and context injection.

Fields

Key Methods

• description (property: str) — Human-readable summary e.g. "DM with Alice", "group: My Group, thread: 12345".
• to_dict() / from_dict() — Serialization round-trip for persistence in sessions.json.

2. SessionEntry — Active Session Record

SessionEntry is the per-session metadata record stored in memory and persisted to {sessions_dir}/sessions.json. Each entry maps a session_key to its current session_id.

Fields

Boolean Flags (State Machine)

SessionEntry has several boolean flags that form a simple state machine governing session behavior on the next access.

State Transition Logic (get_or_create_session)

                    ┌──────────┐
                    │  Incoming │
                    │  Message  │
                    └────┬─────┘
                         │
                         ▼
              ┌──────────────────────┐
              │  session_key exists  │──── No ──► Create fresh SessionEntry
              │  AND !force_new      │
              └──────────┬───────────┘
                         │ Yes
                         ▼
              ┌──────────────────────┐
              │  entry.suspended?    │──── Yes ──► Auto-reset: new session_id
              └──────────┬───────────┘           (reason="suspended")
                         │ No
                         ▼
              ┌──────────────────────┐
              │ entry.resume_pending?│──── Yes ──► Return existing entry
              └──────────┬───────────┘           (preserve session_id)
                         │ No                     Clear flag on next successful turn
                         ▼
              ┌──────────────────────┐
              │   Policy says reset? │──── Yes ──► Auto-reset: new session_id
              └──────────┬───────────┘           (reason="idle"/"daily")
                         │ No
                         ▼
              ┌──────────────────────┐
              │  Return existing     │
              │  entry, bump         │
              │  updated_at          │
              └──────────────────────┘

Priority order in get_or_create_session():

1. suspended=True → always force-reset (hard wipe)
2. resume_pending=True → preserve session_id (soft recovery)
3. No trigger → return existing entry (bump updated_at)

3. SessionStore — Storage and Operations

SessionStore is the main storage layer. It maintains an in-memory dict (_entries) persisted to sessions.json, with SQLite (SessionDB) as the canonical store for session metadata and message transcripts.

Constructor

SessionStore(sessions_dir: Path, config: GatewayConfig, has_active_processes_fn=None)

• sessions_dir — Directory where sessions.json lives.
• config — GatewayConfig instance for routing and housekeeping settings.
• has_active_processes_fn — Optional callback keyed by session_key to check for running background processes. Sessions with active processes are protected from routing-entry pruning.

Operations (Methods)

Internal Helpers

• _ensure_loaded() / _ensure_loaded_locked() — Load sessions.json into _entries dict.
• _save() — Atomic write to sessions.json via temp file + atomic_replace.
• _generate_session_key(source) — Delegates to build_session_key() with config params.

Storage Layout

{sessions_dir}/
  sessions.json          # In-memory _entries dict, persisted as JSON
                           Maps session_key → SessionEntry (metadata only)
  {session_id}.jsonl     # (Legacy, removed in spec 002)

The canonical transcript store is SQLite via SessionDB (from hermes_state). The sessions.json file persists the session_key → session_id mapping and entry metadata (flags, timestamps, token counts). If SQLite is unavailable, the store falls back to JSONL, but this is a degradation path.

4. SessionKey Generation Rules

Session keys are deterministic strings that identify a conversation lane. They are generated by build_session_key(source, group_sessions_per_user, thread_sessions_per_user).

Key Format

agent:main:{platform}:{chat_type}[:{chat_id}][:{thread_id}][:{participant_id}]

DM Rules

• DMs always include chat_id when present, isolating each private conversation.
• thread_id further differentiates threaded DMs within the same DM chat.
• Without chat_id, falls back to user_id_alt or user_id as participant_id.
• Without any identifier, all DMs on that platform collapse to one shared session.

Group/Channel Rules

• chat_id identifies the parent group/channel.
• thread_id differentiates threads within that parent.
• Per-user isolation (append participant_id) is controlled by:
  • group_sessions_per_user (default: True) — group/channel sessions are isolated.
  • thread_sessions_per_user (default: False) — threads are shared by default (Telegram forum topics, Discord threads, Slack threads all share one session per thread).
• participant_id = user_id_alt or user_id (in that priority).
• WhatsApp identifiers are canonicalized to handle JID/LID alias flips.

Special Case: WhatApp

WhatsApp phone numbers go through canonical_whatsapp_identifier() which strips the @s.whatsapp.net suffix and normalizes to E.164 format. This prevents session fragmentation when the bridge returns different alias forms of the same phone number.

5. Multi-User Isolation Strategy

Multi-user isolation determines whether multiple users in the same chat share a conversation or each get their own private session.

Decision Logic (is_shared_multi_user_session)

def is_shared_multi_user_session(source, *, group_sessions_per_user, thread_sessions_per_user):
    if source.chat_type == "dm":
        return False  # DMs are always private
    if source.thread_id:
        return not thread_sessions_per_user  # Threads: shared unless per-user
    return not group_sessions_per_user       # Groups: isolated unless shared

Summary

Impact on System Prompt

When shared_multi_user_session=True, the system prompt omits a fixed user name and instead states: "Multi-user {thread|session} — messages are prefixed with [sender name]. Multiple users may participate." Individual sender names are prefixed on each user message by the gateway at runtime, preserving prompt caching (the system prompt doesn't change per-turn).

6. Explicit Conversation Boundaries

Inactivity and wall-clock time never rotate a conversation. /new and /reset create an explicit boundary; context compression continues to manage long histories. Legacy timer configuration is ignored. The existing SessionResetPolicy datatype is inert compatibility data, not a runtime policy.

Explicit suspension still creates a boundary on the next inbound turn. Recovery respects explicit and historical finalized boundaries rather than reopening them. Resource-only eviction and WebSocket orphan reaping leave conversations resumable.

7. Restart Recovery Flow

The restart recovery system ensures that in-flight sessions are preserved across gateway restarts, crashes, and drain timeouts. It is the solution to issue #7536.

Startup Recovery Sequence

Gateway starts
       │
       ▼
┌───────────────────────────────┐
│ Check for .clean_shutdown     │── Exists? ──► Skip suspension (clean exit)
│ marker                        │
└───────────────────────────────┘
       │ Missing
       ▼
┌───────────────────────────────┐
│ session_store                 │── Marks sessions updated within
│ .suspend_recently_active()    │   last 120 seconds as resume_pending
└───────────────────────────────┘
       │
       ▼
┌───────────────────────────────┐
│ _suspend_stuck_loop_sessions()│── Suspends sessions that have been
│                               │   active across 3+ restarts
└───────────────────────────────┘
       │
       ▼
┌───────────────────────────────┐
│ Queue inbound messages while  │
│ startup restore runs          │
│ (_startup_restore_in_progress)│
└───────────────────────────────┘
       │
       ▼
┌───────────────────────────────┐
│ For each adapter, find        │
│ resume_pending sessions →     │
│ synthesize MessageEvent and   │
│ run _handle_message to let    │
│ the agent auto-continue       │
└───────────────────────────────┘

suspend_recently_active(max_age_seconds=120)

Called on gateway startup when no .clean_shutdown marker exists (indicating a crash or unexpected exit). For each session updated within the last 120 seconds:

• Sets resume_pending=True, resume_reason="restart_interrupted", last_resume_marked_at=now.
• Skips entries already resume_pending=True (no double-mark).
• Skips entries explicitly suspended=True (hard wipe should stay).

Stuck-Loop Detection (_suspend_stuck_loop_sessions)

Counts consecutive restarts via a JSON file ({HERMES_HOME}/restart_counts.json). If a session has been active across 3+ consecutive restarts, it's auto-suspended so the user gets a clean slate.

Drain-Timeout Marking

On graceful shutdown/restart, the drain system calls mark_resume_pending() for any session that was mid-turn when the drain timeout fired. Reasons:

• "restart_timeout" — killed during restart drain
• "shutdown_timeout" — killed during shutdown drain
• "restart_interrupted" — crash recovery (from suspend_recently_active)

All three reasons are in _AUTO_RESUME_REASONS and eligible for startup auto-resume.

Auto-Resume on Next Access

When get_or_create_session() encounters resume_pending=True:

1. It returns the existing entry without creating a new session_id.
2. The existing transcript is loaded intact.
3. The marking is not cleared here — it survives until the next successful turn completes (clear_resume_pending() is called from the gateway after run_conversation() returns a real response).
4. If the resumed turn is interrupted again, the resume_pending flag remains set, and the next restart will retry. The stuck-loop counter handles terminal escalation (3 retries → suspended).

Clean Shutdown Marker (.clean_shutdown)

Written at the end of a graceful shutdown. On next startup:

• If present: skip suspend_recently_active() entirely. Active agents were already drained, so no sessions are stuck.
• Then delete the marker.

This prevents unwanted auto-resets after hermes update, hermes gateway restart, or /restart.

8. Message Queuing Flow

The message queuing system handles two scenarios:

1. Interrupt follow-ups — When a user sends multiple messages while the agent is processing, subsequent messages are queued as single-slot pending messages.
2. /queue FIFO — Explicit /queue commands that must each produce their own full agent turn, in order, without merging.

Data Structures

adapter._pending_messages: Dict[session_key, MessageEvent]
    └── Single "next-up" slot per session. Overwritten on repeat sends
        (burst collapse). Shared with photo-burst follow-ups.

self._queued_events: Dict[session_key, List[MessageEvent]]
    └── Overflow buffer. Each /queue invocation appends here when the
        slot is occupied. Promoted one-at-a-time after each drain.

Enqueue (_enqueue_fifo)

_enqueue_fifo(session_key, event, adapter)
       │
       ▼
┌───────────────────────────────────────┐
│ Is slot free?                         │
│ (session_key NOT in _pending_messages)│── Yes ──► Place event in slot
└───────────────────────────────────────┘
       │ No
       ▼
Append to _queued_events[session_key] (overflow tail)

Dequeue / Promotion (_promote_queued_event)

Called at the drain site after the slot was consumed. If there's an overflow item:

• When pending_event is None (slot was empty), return overflow head as the new event.
• When pending_event exists, stage overflow head in the slot for the next recursion.
• If no adapter available, push back to _queued_events (don't silently drop).

Queue Depth

_queue_depth(session_key, adapter) returns len(overflow) + (1 if slot occupied else 0).

Clearing

Queued events for a session are cleared on /new and /reset (via _handle_reset_command).

FIFO Invariant

Each /queue invocation produces exactly one full agent turn, in FIFO order, with no merging. The single-slot _pending_messages + overflow _queued_events design ensures that repeated sends during an active turn don't cause out-of-order processing.

9. Session Context Injection

SessionContext is built from a SessionSource and GatewayConfig and injected into the agent's system prompt. It tells the agent:

• Where the current message came from
• What platforms are connected
• Where it can deliver scheduled task outputs
• Whether this is a shared multi-user session

Construction (build_session_context)

def build_session_context(source, config, session_entry=None) -> SessionContext

1. Collects connected platforms from config.
2. Collects home channels for each platform.
3. Determines shared_multi_user_session via is_shared_multi_user_session().
4. Attaches session metadata (key, id, timestamps) if session_entry is provided.

PII Redaction (build_session_context_prompt)

The dynamic system prompt section (## Current Session Context) can optionally redact personally identifiable information before sending to the LLM:

• User IDs → user_<12hex> (SHA-256 prefix)
• Chat IDs → <platform>:<12hex> or just <12hex>
• Platforms excluded from redaction: Discord (needs raw IDs for @mentions), and any plugin-registered platform not marked pii_safe.

Redaction applies only to the system prompt text. Routing, session keys, and adapter operations always use the original values.

10. Background Housekeeping

The _session_housekeeping_watcher periodically sweeps idle cached agents, sheds cache entries under memory pressure, and prunes old routing entries hourly. It never ends a transcript because of inactivity or the time of day.

TTL, LRU and pressure eviction commit the live transcript to memory providers before soft-releasing clients. Active turns remain protected; terminal, browser and background process resources survive soft release. Routing-entry pruning preserves the canonical SQLite transcript, and live processes protect their routing entries from pruning. Historical expiry_finalized flags remain recovery fences but are no longer written by a timer watcher.

11. Agent Cache

The gateway maintains an LRU cache of AIAgent instances keyed by session_key to preserve prompt caching across turns.

Cache Properties

• Max size: 128 entries (agent.agent_cache.max_size, default _AGENT_CACHE_MAX_SIZE).
• Eviction policy: Least-recently-used (LRU via OrderedDict).
• Idle TTL: 3600s (1h) — agent.agent_cache.idle_ttl_secs, enforced by _session_housekeeping_watcher.
• Memory budget: agent.agent_cache.memory_high_mb (default auto) — see below.
• Lock: _agent_cache_lock (threading) for thread safety.

Memory-Pressure Eviction

A cached agent pins _session_messages, the full live transcript including tool outputs — tens of MB on a session with 100+ tool calls. The entry cap and the idle TTL are both blind to that: a gateway serving many chats keeps every warm transcript resident (agents that took a turn within the TTL are never idle-swept), so RSS climbs until the cgroup throttles and SIGTERM can no longer flush inside systemd's stop timeout (#80764).

_sweep_agent_cache_under_pressure() is the valve. Each watcher tick it compares the process's anonymous RSS against memory_high_mb; over budget, it evicts LRU agents through the same soft path the cap enforcer uses (_commit_then_release_soft), then runs malloc_trim so the freed arenas actually return to the OS. Evicted sessions rebuild their transcript from the persisted session on the next turn.

Three classes of session are never shed:

• agents currently mid-turn (their clients and sandboxes are in use);
• the protect_recent most-recently-used sessions (their prompt cache is worth the most);
• any session whose live transcript has not finished reaching disk — transcript_persistence_caught_up() compares _last_flushed_db_idx against len(_session_messages), the same divergence the FTS write-corruption guard reacts to when it preserves live history over a lagging transcript.

memory_high_mb: auto derives the budget from the cgroup limit the gateway runs under (memory.high, then memory.max, then cgroup v1), falling back to total RAM when uncapped. Set a number to pin it, or 0/off to disable the pass entirely. Helpers live in gateway/agent_cache_pressure.py.

Cache Lifecycle

Message arrives
    │
    ▼
get_or_create_session()  →  session_key obtained
    │
    ▼
Lookup _agent_cache[session_key]
    │
    ├── Hit → move_to_end(), reuse AIAgent (preserves prompt cache)
    │
    └── Miss → create new AIAgent, store in cache
                (if at capacity, popitem(last=False) evicts LRU entry)
    │
    ▼
run_conversation()  →  agent processes message
    │
    ▼
Housekeeping soft-releases idle agents without ending transcripts

Cleanup Flow

Resource eviction removes the cached agent and commits memory before soft-releasing clients. Full _cleanup_agent_resources(agent) teardown is reserved for actual conversation boundaries and shutdown.

Appendix: Key Configuration

State database and FTS recovery

The canonical transcript lives in the sessions and messages tables. FTS5 tables and their sync triggers are derived indexes that can be detached and rebuilt without deleting canonical messages. See docs/state-db-recovery.md for the bounded live failure mode and the explicit repair procedure.

Conversation lifetime

No idle or daily reset settings are supported. Explicit /new and /reset, compaction, suspension and crash recovery retain their separate lifecycle roles.