.agents/skills/agent-testing/references/report.md
Every automated test session ends with a structured, evidence-backed report. A chat-only summary is not an acceptable deliverable: the report is what the user (or a reviewer, or a later agent) audits without replaying the session.
Reports live under .records/reports/ (gitignored, like all .records/
output), grouped by acceptance subject. One subject directory contains one
subdirectory per immutable verification round:
.records/reports/<subject-key>/
├── acceptance.json
├── <YYYYMMDD-HHMMSS>-<slug>/
│ ├── report.md
│ ├── result.json
│ └── assets/
└── <YYYYMMDD-HHMMSS>-<slug>/
<subject-key> is the ingest subject with : replaced by -. Scaffold with
report-init.sh --subject topic:tpc_xxx <slug> "<title>"; this also pre-fills
result.json.subject. The legacy flat layout remains readable, but new runs
should always carry their subject.
Reusable per-check inputs live separately under
.records/fixtures/<subject-key>/<check-id>/ as check.json plus seed/.
Execution outputs remain in the round directory's assets/. See
scripts/fixture.mjs and the skill's fixture workflow.
result.json is the report — report.md is just its tail. The published
acceptance page renders itself from result.json: one line of
provenance (PR / branch / commit / date / surfaces), the overall conclusion from
summary.conclusion directly under the title, and the check list from plan[]
paired with cases[]. So report.md must NOT repeat the scope block or a case
table — those double up on the page. It carries only the non-duplicate narrative
(follow-ups / this-round notes / score), rendered as the page's collapsible
"Details".
Scaffold up front — before running the first test step:
# $SKILL_DIR = the skill's install dir
DIR=$("$SKILL_DIR/scripts/report-init.sh" --subject topic:tpc_xxx my-slug "My title")
The script creates the directory, pre-fills branch / commit / date in both files, and prints the directory path. Translate its headings and table labels to the user's language before delivery if needed.
Collect evidence as you test — every asserted behavior gets one evidence
item in $DIR/assets/:
UI (static state): agent-browser screenshot or capture-app-window.sh, then
verify the screenshot with the Read tool before citing it — never cite an
image you haven't looked at.
UI (time-based behavior): screenshot vs GIF is a judgment you make per case. If the assertion is about change over time — streaming output, a ticking timer, loading/progress states, animations, appear/disappear transitions — a static screenshot cannot prove it. Record a frame sequence and synthesize a GIF:
# start recording (background), trigger the behavior, wait for it to finish
"$SKILL_DIR/scripts/record-gif.sh" "$DIR/assets/case2-streaming.gif" 12 2 &
GIF_PID=$!
# ... drive the scenario ...
wait $GIF_PID
Verify at least the first/last frames visually (Read the GIF) before citing.
UI (before/after comparison): capture and visually verify both original
screenshots. Do not compose them into a new image. In the case's evidence
array, pair them with a shared comparison id.
A comparison pair means the same view in two states. Sequential steps of a flow are not before/after states; attach those as ordinary ordered evidence items with captions naming each step.
"evidence": [
{
"path": "assets/before.png",
"comparison": { "id": "topic-row", "role": "before", "layout": "vertical", "label": "before: 11px, line-height 40px" }
},
{
"path": "assets/after.png",
"comparison": { "id": "topic-row", "role": "after", "layout": "vertical", "label": "after: 12px, line-height 44px" }
}
]
comparison is a nested object on each half. Writing it flat —
{ "path": "…", "comparison": "topic-row", "role": "before" } — is the
usual slip, and it does not pair: role is read from inside comparison,
never from the evidence item itself.
The acceptance page renders a complete pair with each screenshot under its own
tinted band — red for before, green for after. A group contains exactly one
before and one after, and both halves need the same string id; a half
without an id can never pair. Incomplete groups render as ordinary evidence.
acceptance run ingest warns on every malformed comparison it drops —
treat that warning as a failed publish and re-ingest a corrected round,
exactly as with a skipped evidence upload.
Two fields are worth setting on every pair:
layout — horizontal (default, side by side) or vertical (stacked).
A tall, narrow crop (a sidebar, a form, a list) reads well side by side; a
wide, short strip (a toolbar, a one-line footer) must be vertical,
because two of them in a two-column grid become illegible slivers. Set it on
both halves.label — the caption shown next to the role word in the band. This is
where the before/after contrast is actually stated: put the measured delta
on each side, so the two captions read as a comparison rather than repeating
the case title.Audio (a deliverable the user hears — TTS output, a voice reply, an
alert tone): attach the clip itself as audio evidence. The acceptance page
renders a player, so the reviewer can listen; prose about a sound, or a
screenshot of a waveform, proves nothing. mp3 / wav / m4a / aac /
flac / ogg / opus are typed as audio from the extension.
Verify the clip before citing it — confirm it is non-silent and carries the expected content (duration plus a transcription or spectral check); an empty or truncated file is indistinguishable from a good one in a file listing. Pair it with a short text artifact when the claim is about what was said (input text, voice/model, measured duration): the player proves it plays, the text makes it auditable.
A screen recording with system audio is the fallback for "the UI plays it at the right moment"; for "the output is correct", attach the file the feature produced.
CLI: use the dual-text evidence format below. Preserve the exact command +
trimmed output (<cli> <command> | tee "$DIR/assets/x-execution.txt") in
the execution artifact, and attach a separate reasoning artifact.
Network: agent-browser network requests dumps or HAR files.
For CLI, API, backend, policy, security, migration, and other non-visual behavioral checks, one text file rarely serves both audiences well. A reviewer needs to understand why the check is meaningful; an auditor needs the concrete observations. Attach two separate text artifacts to the same case, in this order:
<check>-reasoning.md) — concise, reviewer-facing:
<check>-execution.txt or .md) — audit-facing:
The split is semantic, not cosmetic. Do not duplicate the same prose into both files, and do not turn the execution artifact into a second high-level summary. Trim unrelated noise, but retain values that make the outcome independently auditable (for example exit codes, error text, file existence, response status, or server receive counts).
{
"evidence": ["assets/write-boundary-reasoning.md", "assets/write-boundary-execution.txt"],
"id": "write-boundary",
"name": "approved writes succeed and escape attempts are denied",
"observation": "control exit=0; four escape attempts exit non-zero and created no files",
"status": "pass"
}
This is a hard default for non-visual behavioral claims, with two narrow exceptions:
If a follow-up round corrects either half, publish both halves again in that round. Every immutable round must be a self-contained decision snapshot; never ask the reviewer to combine reasoning from an older round with execution logs from the current one.
Write plan[] BEFORE you run anything. The approved plan from Step 1 is part
of the report, not scaffolding you throw away: each item is
{ id, title, category, verifier, method, expected, requiredEvidence } — what
you will check, which requirement area it belongs to, how it is judged, how you
will exercise it, what would make it pass, and the artifact it must produce.
verifier and requiredEvidence are closed sets the pipeline acts on (schema
below); method / expected are prose. cases[] later reuses the same ids,
which is what lets the report pair intent against outcome. A planned item that
never produces a case renders as 未执行 rather than vanishing, so cut coverage
in the open.
HARD RULE: no programmatic gates in the plan. Tests / type-check / lint / build are never plan items — ingest drops them and a gates-only round fails to publish. See what is NOT an acceptance check.
Fill result.json as you go — it is the report. Each tested behavior is one
entry in cases[] ({ id, name, result, observation, evidence }), where
evidence is a path under assets/. Set the scope fields (scenario, branch,
commit, surfaces, entry) and write the one-paragraph verdict into
summary.conclusion. The page pairs each check with its evidence inline, so you
don't hand-build a table. report.md holds only the narrative tail.
scenario is a closed enum, not a description — see the table below. The
scaffold pre-fills coding; a one-line summary of what the run covers belongs in
context, never in scenario.
Set the verdict in both report.md and result.json. Describe key visual
outcomes in prose; the published acceptance URL is the only visual pointer in
the final chat reply.
Publish (SKILL.md Step 6) — upload the finished session so it's viewable on
LobeHub Acceptance, not just on disk. Publish to PRODUCTION defaults with the
user's real login, NOT a local-dev CLI override — strip the local dev overrides
so lh uses its production defaults. Clearing an override profile looks like:
env -u LOBEHUB_SERVER -u LOBE_API_KEY -u LOBEHUB_CLI_API_KEY -u LOBEHUB_CLI_HOME \
lh acceptance run ingest "$DIR" --source agent-testing --open --json
This creates a new immutable verification run, attaches it to the required
subject acceptance, uploads the cases, evidence, and report body, then prints
/acceptance/<acceptanceId> plus its ?r=<roundIndex> round-snapshot form.
Include only the full production acceptance link in the final reply. Never
expose local paths, local file links, or internal run-page paths. Leave
whitespace after the URL, so
an autolinker can't swallow adjacent CJK punctuation into the href. See SKILL.md →
Step 6 for why production defaults (a localhost URL isn't shareable and a local
stub storage fails file-evidence uploads), the production login check, and the
atomic commands (acceptance run … (plus … result, … evidence, … report)).
report.md MUST be written in the language the user is conversing in — the
whole file, headings included. If the conversation is in Chinese, the report is in
Chinese; do not mix English prose into it. The scaffold headings are placeholders —
translate them when filling. Exceptions that stay as-is: code/commands,
identifiers, log excerpts, and result.json (its keys and status values are
machine-read and stay English; the title and case name fields follow the
user's language).
Default report shape (a case table doubles the page and is only for a purely
non-visual run; for UI runs, leave the case list to result.json):
| Section | Content |
|---|---|
| Verdict | Overall verdict first (pass / partial / fail), then concise reasons and follow-ups |
| Verification | Commands or automated checks run in this session, with trimmed results |
| Score | Pass/fail/blocked counts, optional 0–100 score |
Status values: pass / fail / blocked (couldn't run — e.g. auth or env
missing; a blocked case is not a pass).
Two fields are the report's identity in every list surface — treat them as REQUIRED on every ingest:
title (top level) — without it the run lists as "未命名验证" forever.summary.verdict (pass / fail / partial) — without it the list glyph is a
permanent amber "?" instead of the green pass. The CLI derives a fallback from the
cases, but an explicit verdict is still the author's job.comparison pair side should carry a label — the role band renders it as
the explanation; a pair without labels shows two bare role words and reads as
unexplained.{
"branch": "feat/task-tree",
"cases": [
{
"category": "Task hierarchy",
"id": "1",
"name": "task tree returns nested children",
"surface": "cli",
"status": "pass",
"observation": "root returned 3 nested children, depth 2",
"evidence": ["assets/task-tree.txt"]
},
{
"category": "Tab responsiveness",
"id": "2",
"name": "conversation tab switching avoids duplicate parsing",
"surface": "desktop",
"status": "pass",
"observation": "The switch-time parsing hotspot disappeared and GC time fell.",
"evidence": ["assets/benchmark.json", "assets/cpu-profile.json"],
"datasets": [
{
"id": "switch-metrics",
"fields": [
{ "key": "name", "type": "string" },
{ "key": "before", "type": "number", "unit": "ms" },
{ "key": "after", "type": "number", "unit": "ms" },
{ "key": "direction", "type": "category" },
{ "key": "target", "type": "number", "unit": "ms" }
],
"rows": [
{
"name": "GC self-time",
"before": 257,
"after": 24.8,
"direction": "lower",
"target": 50
}
]
}
],
"visualizations": [
{
"id": "switch-comparison",
"type": "metric-comparison",
"version": 1,
"dataset": "switch-metrics",
"title": "Performance comparison",
"context": "Electron 40, warm cache, identical tool-heavy topic fixture",
"encoding": {
"label": "name",
"before": "before",
"after": "after",
"direction": "direction",
"target": "target"
}
}
]
}
],
"commit": "abc1234",
"context": "Nested task tree API behind the new repository method",
"createdAt": "2026-06-11T15:30:00+08:00",
"entry": "<cli> task list --tree",
"interactionCost": {
"model": "goms-klm@lobe-v1",
"scope": "user-equivalent",
"totalSeconds": 8.1,
"activeSeconds": 6.1,
"waitSeconds": 2,
"operators": { "K": 1, "P": 2, "H": 0, "M": 2, "T_chars": 5, "R_ms": 2000 },
"phases": []
},
"plan": [
{
"id": "1",
"title": "task tree returns nested children",
"category": "Task hierarchy",
"verifier": "program",
"method": "<cli> task list --tree against a 3-level fixture",
"expected": "root shows 3 nested children at depth 2",
"requiredEvidence": ["text"]
},
{
"id": "2",
"title": "conversation tab switching avoids duplicate parsing",
"category": "Tab responsiveness",
"surface": "desktop",
"verifier": "program",
"method": "Run the same warm-cache CDP switch profile before and after the change",
"expected": "GC self-time is at or below 50 ms",
"requiredEvidence": ["text"]
}
],
"pullRequest": {
"number": 17152,
"title": "feat(task): nested task tree",
"url": "https://github.com/<org>/<repo>/pull/17152"
},
"scenario": "coding",
"summary": {
"total": 2,
"passed": 2,
"failed": 0,
"blocked": 0,
"score": 100,
"verdict": "pass"
},
"surfaces": ["cli"],
"title": "Verify task tree API"
}
plan[] is the checks you committed to before running them, and it shares
ids with cases[]. Every plan item must carry a category that names its
user-facing business scenario or requirement area (for example Task hierarchy,
Rate-limit recovery, Browser actions). It must not name a technical surface such
as Desktop, CLI, or Backend: Acceptance groups are organized by what the user
is accepting, while surface separately records where the check ran. A plan item
with no matching case renders as 未执行: cutting coverage is allowed, hiding that
you cut it is not.
Two of its fields are a closed vocabulary, because the pipeline acts on them — they are not labels:
| field | values | what it does |
|---|---|---|
verifier | program | agent | llm (default agent) | How the verdict is reached. A command-asserted check is program; calling it agent hides what actually judged it. |
requiredEvidence | screenshot | gif | video | audio | text | markdown | dom_snapshot | transcript | The artifact this check must produce. The coverage gate fails an item whose required medium is missing. |
An out-of-vocabulary value in either fails the ingest — an unrecognized medium would silently gate on nothing, which is worse than no gate at all.
method (how you would exercise it) and expected (what would make it pass) stay
free prose — they carry intent no enum can, and both render under the check on
the page next to the outcome.
A check is something a person decides about the delivery. The repo's own automated gates are not that, and on the page they are actively harmful: twenty green "unit tests pass" rows bury the two checks that actually needed someone to look.
These MUST NOT appear in plan[] / cases[], under any phrasing:
| Not a check | Where it belongs |
|---|---|
| Unit / integration / regression / snapshot tests, coverage | one line in report.md → Verification |
type-check, tsc, eslint, lint, format, a clean build | same — a precondition of shipping, not a deliverable |
| "the suite is green", "CI passes" | same |
This is enforced, not advisory. acceptance run ingest drops every matching
item — matched on title, category, AND method, so "run bun run test"
inside method under a product-sounding title still matches — warns with the
dropped ids, and recounts summary from the checks that remain. A round
consisting only of such checks fails to publish. Apply the rule at plan
time (Phase 1 case selection gate), before any case runs: a gate written as a
check is a round spent proving something nobody accepts. Run the gates as your
own diligence — report them as one line of narrative.
The line is the subject of the check, not who judged it: a CLI behavior check
asserted by a command is a good acceptance item (verifier: "program");
"bun run test is green" is not.
What IS a check: what the user sees, hears, reads, or receives — a rendered screen, a produced file, a response shape a client depends on, an audio clip that actually plays, a failure state that recovers.
A plan item may also carry a per-item surface (same closed set as the run-level
surfaces; electron normalizes to desktop). It says which product surface THIS
check ran on. It is metadata, never an Acceptance grouping key.
surfaces is a closed set — web | desktop | cli | mobile | bot — and
names the product surface a check ran on. electron is accepted and normalized
to desktop. Anything else fails the ingest:
unit, backend, database, type-check do
not belong here; a backend change verified through the CLI has surface cli.method.entry is the command or URL exercised (<cli> task list --tree, /chat/settings)
— not a PR title and not a description of the change.
scenario is a closed set — coding | writing | research | generic —
naming what KIND of delivery was verified, because the page renders a different
scope header for each. It defaults to coding when omitted, and an out-of-set value
fails the ingest rather than being stored:
| value | the delivery under verification |
|---|---|
coding | a software change (branch / commit / surfaces under test) |
writing | a written deliverable (manuscript / chapters / documents) |
research | a research deliverable (question / sources / claims) |
generic | anything else — no modeled scope; context is an open bag |
It is not a free-text summary of the run. Writing the sentence you would say
out loud ("verify the memory tool renders…") is the easy mistake — the scaffold
pre-fills coding, so overwriting it with prose turns a working file into a hard
ingest failure at the very last step. That sentence belongs in context, which is
the scenario's own scope bag and is rendered next to scenario in the page's scope
header; for a non-coding scenario it also carries that scenario's modeled fields.
A case may provide datasets[] plus visualizations[]. The ingest stores the
versioned manifest on the check result and the Acceptance page renders each view.
The first supported renderers are metric-comparison, line-chart, bar-chart,
scatter-plot, heatmap, and table (all version: 1). Each view references one dataset by id
and maps its fields through encoding.
Use line-chart.encoding.series[].style (muted | primary | accent) to keep a
baseline visually quiet and emphasize the compared run. Tables can mark best-in-column
values with encoding.highlights[] ({ field, mode: "min" | "max" }); ties are all
marked SOTA. bar-chart is the default for grouped model or benchmark score comparisons.
Inline datasets use declared fields[] and object rows[]; undeclared cells,
unsupported renderers, missing dataset references, and more than 10,000 total
rows fail ingest. Use inline data only for a review-sized summary. Keep raw
benchmark output, traces, vectors, or profiles in evidence; visualization is a
decision aid, not a replacement for evidence.
For comparable metrics, only publish before/after values produced by the same harness, fixture, environment, warm-up policy, statistic, and sample window. If those differ, use separate series or mark the case uncertain instead of presenting a misleading delta.
pullRequest is optional: when absent, the ingest asks gh for the PR of branch
and fills it in. Write it explicitly only when the report verifies a PR that isn't
the branch's own.
score is optional — use it when the verdict has a subjective component (UI polish,
copy quality); omit it for purely binary runs. verdict is the single word the user
reads first: pass, fail, or partial.
subject identifies the business subject whose acceptance aggregate owns this
immutable run: either "subject": "task:<id>" (task | topic | document) or
{ "type": "task", "id": "task_…", "requirement": "one-sentence acceptance bar" }.
The --subject flag overrides this field.
An Operation ID is not part of report identity and is not required for external
agent-testing. acceptance run ingest creates a standalone Verify Run. Use
--operation only to link the report to a real, existing LobeHub Agent Run under
test. For atomic publication, carry the verifyRunId returned by
acceptance run create and pass it through --run; never ask an external-project
user to supply an Operation ID or fabricate one.
Choose by continuity: use the current Topic for work discussed and iterated in that conversation; use a Task only when it already owns the deliverable or the work intentionally needs independent, durable, cross-topic tracking; use a Document only when the document itself is under acceptance. Create a Task only when no relevant subject exists. A terminal Acceptance may require a new Acceptance, but it does not by itself justify changing the subject from Topic to Task.
Inside a relevant LobeHub conversation, both subject fields may be omitted because
acceptance run ingest defaults to topic:$LOBEHUB_TOPIC_ID; outside a topic, an
explicit subject is mandatory. Every ingest creates a new immutable run; never
update a prior run after a fix, publish the re-verification as the next round.
interactionCost is optional and run-level. For UI runs driven through
agent-browser, create interaction-trace.jsonl with scripts/agent-browser-klm.mjs,
then run scripts/agent-browser-klm-analyze.mjs --trace "$DIR/interaction-trace.jsonl" --result "$DIR/result.json" --write.
When published, acceptance run ingest maps each case onto a check result:
name→title, status/result→verdict, observation→the result's key
observation, and evidence paths→uploaded artifacts. summary.{total,passed,failed,blocked}
and verdict become the report's stats + overall verdict; report.md becomes the
report body.
pass/fail in cases[] must link at least
one asset. UI cases must attach their primary screenshot/GIF as evidence;
non-visual behavioral cases must attach both reasoning and execution text;
transcripts, HAR files, and logs belong in the execution half.