Eval Directory

This directory contains prompts for various software engineering tasks that can be executed by an agent. Each subdirectory represents a single, self-contained task.

These are intended to be used as an eval set for regression testing on a Chromium CI builder in addition to being a reference for new tasks.

Subdirectory Structure

Each subdirectory should contain all the necessary files and information for the agent to perform the task. This includes:

• eval.md: A file describing the task and its outcome.
• prompt.md: The prompt that initiates the task.

README.md Format

Note: this format is subject to change as a regression test suite is implemented.

The eval.md file in each subdirectory should follow this format:

• Owner: List or person to contact if the task stops reproducing.
• Description: A brief description of the task.
• Git-Revision: The git revision on which the task was successfully performed. This is used for reproducibility.
• Result: A summary of what the agent accomplished.
• Modified files: A list of the files that were modified by the agent during the task.

This structure ensures that each task is well-documented and can be easily understood and reproduced.