Memories Pipeline (Core)

This module runs a startup memory pipeline for eligible sessions.

Prompt Templates

Memory prompt templates live under codex-rs/core/templates/memories/.

• The undated template files are the canonical latest versions used at runtime:
  • stage_one_system.md
  • stage_one_input.md
  • consolidation.md
  • read_path.md
• In codex, edit those undated template files in place.
• The dated snapshot-copy workflow is used in the separate openai/project/agent_memory/write harness repo, not here.

When it runs

The pipeline is triggered when a root session starts, and only if:

• the session is not ephemeral
• the memory feature is enabled
• the session is not a sub-agent session
• the state DB is available

It runs asynchronously in the background and executes two phases in order: Phase 1, then Phase 2.

Phase 1: Rollout Extraction (per-thread)

Phase 1 finds recent eligible rollouts and extracts a structured memory from each one.

Eligible rollouts are selected from the state DB using startup claim rules. In practice this means the pipeline only considers rollouts that are:

• from allowed interactive session sources
• within the configured age window
• idle long enough (to avoid summarizing still-active/fresh rollouts)
• not already owned by another in-flight phase-1 worker
• within startup scan/claim limits (bounded work per startup)

What it does:

• claims a bounded set of rollout jobs from the state DB (startup claim)
• filters rollout content down to memory-relevant response items
• sends each rollout to a model (in parallel, with a concurrency cap)
• expects structured output containing:
  • a detailed raw_memory
  • a compact rollout_summary
  • an optional rollout_slug
• redacts secrets from the generated memory fields
• stores successful outputs back into the state DB as stage-1 outputs

Concurrency / coordination:

• Phase 1 runs multiple extraction jobs in parallel (with a fixed concurrency cap) so startup memory generation can process several rollouts at once.
• Each job is leased/claimed in the state DB before processing, which prevents duplicate work across concurrent workers/startups.
• Failed jobs are marked with retry backoff, so they are retried later instead of hot-looping.

Job outcomes:

• succeeded (memory produced)
• succeeded_no_output (valid run but nothing useful generated)
• failed (with retry backoff/lease handling in DB)

Phase 1 is the stage that turns individual rollouts into DB-backed memory records.

Phase 2: Global Consolidation

Phase 2 consolidates the latest stage-1 outputs into the filesystem memory artifacts and then runs a dedicated consolidation agent.

What it does:

• claims a single global phase-2 job (so only one consolidation runs at a time)
• loads a bounded set of stage-1 outputs from the state DB using phase-2 selection rules:
  • ignores memories whose last_usage falls outside the configured max_unused_days window
  • for memories with no last_usage, falls back to generated_at so fresh never-used memories can still be selected
  • ranks eligible memories by usage_count first, then by the most recent last_usage / generated_at
• computes a completion watermark from the claimed watermark + newest input timestamps
• syncs local memory artifacts under the memories root:
  • raw_memories.md (merged raw memories, latest first)
  • rollout_summaries/ (one summary file per retained rollout)
• prunes stale rollout summaries that are no longer retained
• if there are no inputs, marks the job successful and exits

If there is input, it then:

• spawns an internal consolidation sub-agent
• builds the Phase 2 prompt with a diff of the current Phase 1 input selection versus the last successful Phase 2 selection (added, retained, removed)
• runs it with no approvals, no network, and local write access only
• disables collab for that agent (to prevent recursive delegation)
• watches the agent status and heartbeats the global job lease while it runs
• marks the phase-2 job success/failure in the state DB when the agent finishes

Selection diff behavior:

• successful Phase 2 runs mark the exact stage-1 snapshots they consumed with selected_for_phase2 = 1 and persist the matching selected_for_phase2_source_updated_at
• Phase 1 upserts preserve the previous selected_for_phase2 baseline until the next successful Phase 2 run rewrites it
• the next Phase 2 run compares the current top-N stage-1 inputs against that prior snapshot selection to label inputs as added or retained; a refreshed thread stays added until Phase 2 successfully selects its newer snapshot
• rows that were previously selected but still exist outside the current top-N selection are surfaced as removed
• before the agent starts, local rollout_summaries/ and raw_memories.md keep the union of the current selection and the previous successful selection, so removed-thread evidence stays available during forgetting

Watermark behavior:

• The global phase-2 job claim includes an input watermark representing the latest input timestamp known when the job was claimed.
• Phase 2 recomputes a new_watermark using the max of:
  • the claimed watermark
  • the newest source_updated_at timestamp in the stage-1 inputs it actually loaded
• On success, Phase 2 stores that completion watermark in the DB.
• This lets later phase-2 runs know whether new stage-1 data arrived since the last successful consolidation (dirty vs not dirty), while also avoiding moving the watermark backwards.

In practice, this phase is responsible for refreshing the on-disk memory workspace and producing/updating the higher-level consolidated memory outputs.

Why it is split into two phases

• Phase 1 scales across many rollouts and produces normalized per-rollout memory records.
• Phase 2 serializes global consolidation so the shared memory artifacts are updated safely and consistently.