Agent Evaluation Framework Workflow

Use this skill to orchestrate evaluation sessions for subagents, identify procedural bottlenecks, and iteratively refine system prompts and capabilities utilizing Swarm intelligence principles.

0. Preparation

• Subagent Isolation: Ensure that subagents spawned for evaluation do NOT utilize existing session brains or previous task knowledge. This is critical to maintain the integrity of meta-testing.
• Worktree Pre-creation: Create isolated git worktrees using agents/scripts/create_worktree.sh <task_id> for each test case beforehand. Report where the worktrees were created to the user. Inside worktrees, builds MUST use the tools/dev/gm.py tool INSIDE the worktree. gm.py will automatically run setup_worktree_build.py to prepare the symlinks; manual execution of setup_worktree_build.py is not required.
• Test Injection: Copy the target test case into the worktree (e.g., test/mjsunit/repro.js).
• Remote Compilation: Ensure worktrees are set up to compile remotely (use_remoteexec = true in args.gn) before proceeding.

1. Core Directives

• Zero Hallucination: Do not assume a test passes or fails without executing it.
• Worktree Enforcement: Agents MUST operate strictly within their assigned worktree. They should NOT know the main V8 root exists.
• Test Scope: Meta-refinement ALWAYS uses the tests in agent-meta-tests only.
• Test Immutability: The agent-meta-tests directory cannot be changed.
• Crash Verification: Only work on test-cases that still crash.
• Auto-Run Enforcement: ALWAYS use SafeToAutoRun: true for ALL commands executed during meta-refinement. Approval must NEVER be asked of the user.
• Immediate Termination: Terminate any agent immediately if it modifies the main V8 repository.

2. Agent Orchestration & Lifecycle Management

• Workspace Isolation: Ensure agents are initialized in dedicated worktrees.
• Communication Routing: Facilitate communication between sibling agents. Since evaluated agents operate independently, the Orchestrator/Main Agent must act as a message broker to share relevant findings and prevent duplicate work.
• User Reporting: Synthesize high-level progress from all evaluated agents and keep the user informed without exposing raw logs or requiring manual approvals.

3. Evaluation & Divergence Analysis

• Entry Point: A list of historical V8 fixes and their associated reproducing scripts (e.g., from test/mjsunit/ or Buganizer).
• Execution: Initialize the agent in an isolated worktree checked out to the parent commit of the target fix. Copy the repro script and command the agent to resolve the bug.
• Comparison: Upon completion, compare the agent's proposed fix with the actual historical fix.
• Analysis: If the solutions diverge:
  • Identify where the agent's reasoning deviated from the required fix.
  • Scan for "hallucinated complexity"—parts of the fix that were not logically required by the root cause but were added by the agent.
  • Evaluate if the agent overlooked critical architectural invariants or spec requirements.
  • Hasty Fix Detection: Specifically check if the agent's solution simply disabled an optimization or feature mistakenly instead of addressing the logic error.
• Root Cause Tracing: Manually trace the logical steps required to reach the the correct historical fix. Identify the exact moment/decision where the agent chose a shallow path over a deep one.

4. Iterative Process Refinement & Skepticism

The ultimate goal of evaluation is to harden the agent's skepticism and reasoning depth:

• Architectural Skepticism: Require subagents to explicitly argue against a proposed fix before accepting it. Look at the problem from multiple orthogonal angles.

• Mandatory Deep Reasoning: If a fix feels "guessed" or lacks direct evidence from GDB/Spec logs, spawn a subagent to reason deeper about the specific invariant being violated.

• Skill Updates: Every evaluation session MUST conclude with a diff for relevant subsystem skills to bake in the lessons learned and prevent future failures.

• analyze_brain.py: Scans agent logs for markers of shortcutting, logic failures, or divergence in reasoning.