.agents/shared/test-loop.md
The portable core of autonomous, tested implementation. perform-task wrappers own repository setup, commit boundaries, account safety, and host-specific drivers. This protocol owns task-derived evidence selection, execution, assessment, recovery, and reporting.
No instrument is globally mandatory. Its prerequisites become mandatory only after assessment selects a check that needs it.
EVIDENCE_AUTHOR -> RUN_SELECTED_CHECKS -> ASSESS
APPROVED:
every selected check has direct positive evidence; return DONE.
TEST_FLAW:
the command, environment, fixture, probe, oracle, control, capture or evidence
path could not decide the claim; keep the implementation and repair only the
invalid checks.
IMPL_BUG:
a sound check exposed a defect; fix the implementation, run targeted general
and affected-specialist review, retain the next implementation attempt, then
rerun only invalidated checks.
UNRECOVERABLE:
a required subject or capability cannot be reached safely after independent
directness assessment; preserve exact evidence and return BLOCKED.
One run is one planned execution set under a unique run directory. It may contain several related commands or checks and may mix instruments. Do not pack unrelated commands into one shell invocation merely to reduce the run count. Carry passing evidence forward; a rerun executes only failed or invalidated checks.
Select the most direct practical instrument that could detect the negative. Cheaper is better only when it still executes the changed surface.
For every command record exact command line, working directory, environment additions, exit code and complete local log. Direct by-products to the ignored run directory or an existing ignored build tree. For every artifact or absence claim, quote literal readings and include a known-present control when a typo could otherwise pass.
Use Telegram only when it adds causal coverage. App behavior that lives in the client normally requires the Debug binary and instrumented execution. Visible claims additionally require captures. Isolated library or build behavior may be better proved by a small binary or consumer build without Telegram. Deletion uses controlled absence, regeneration, affected builds and retained tests; neighboring network or UI campaigns do not belong unless reverting the deletion could change their result.
A TEST_FLAW rerun must eliminate an assumption or move closer to the changed surface. Record before each recovery:
When the flaw's cause is the instrument idiom rather than this task's fixture,
repair the harness in the same run, not only the overlay: add the missing
helper to Telegram/SourceFiles/test/, tighten an existing contract so the
failed technique becomes unreachable, or add the symptom to that directory's
README failure-diagnosis table. Prefer making the technique impossible over
documenting that it is forbidden — the repair that retired the unbounded probe
accessor also exposed a check that had been passing vacuously in two earlier
runs, which a written rule would not have caught. A diagnosis left in one
task's notes is rediscovered by the next task at full campaign cost, and this
is the mechanism that stops it.
Before writing any local overlay helper, search that directory first. An overlay that reimplements a shared facility is itself a TEST_FLAW risk: the local copy carries none of the refusals the shared one accumulated.
The directness ladder is task-dependent:
After a repeated signature, use a fresh recovery assessment and forbid the failed technique. MAX_TEST_RUNS closes one campaign, not the task: preserve passes, isolate unmet checks, and either start focused recovery with a different instrument or prove every applicable strategy unsafe, unavailable, or non-causal in a Recovery exhaustion table. Time spent, the cap, repeated failure, probe complexity, and a blank screenshot are not exhaustion.
Default to not approved. A check passes only on positive recorded evidence compared with its declared oracle.
The performer owns source commits. Evidence scripts, probes, logs and overlays stay under task-local work, evidence, or ignored run storage unless the task explicitly retains a permanent test. Never stash. test.md is the only implementation-fix handoff and names the failing check, root cause, evidence and fix hint.
Retained source commits use one normal subject, a blank line, and Task: <TASK_ID>, with the conditional [ai] prefix required by repository rules. Attempt numbers never enter commit messages.
This entire section applies only when the evidence design selects a Telegram application launch. Direct readings, commands, artifact checks, unit suites, probes, and non-Telegram component binaries do not require or touch portable data.
The debug build runs in portable mode out of out/Debug/. Three sibling folders matter:
test_TelegramForcePortable — the golden test account, prepared by the user. Read-only SOURCE,
never modified by tests. (Its presence is the launch gate; the wrapper aborts if it is missing.)TelegramForcePortable — the LIVE folder the app actually uses (its presence is what puts the
build in portable mode). A marker file named testing directly inside it marks it as a
disposable test copy; a live folder WITHOUT the marker is the user's real data.real_TelegramForcePortable — the user's real data, preserved so manual use survives. Once it
exists, NO flow step may ever delete, rename, move, overwrite, or write into it.SETUP — run at the START of every test run, with NO app instance alive. It is idempotent: the
first SETUP after a crash moves leftover crash files and can refuse before launch; after successful
relocation, the next SETUP finds nothing left to move.
The workspace helper's test-run command performs exactly these steps before every launch, and
test-account-reset performs the broken-account recovery below; the manual steps remain the
contract those commands implement.
test_TelegramForcePortable. Its absence is a portable-account setup blocker.TelegramForcePortable/testing exists, the live folder is already the reusable test copy:
never copy, move, or delete any of the three folders. Clear only what an earlier run left
inside the live copy — move a non-empty TelegramForcePortable/tdata/working into
<EVIDENCE_DIR>/stale-crash/ and every TelegramForcePortable/tdata/dumps/*.dmp into
<EVIDENCE_DIR>/stale-crash/dumps/, then proceed straight to testing. <EVIDENCE_DIR> must be
a run-specific directory the repository ignores, never a tracked one: a preserved minidump
is routinely tens of megabytes, and a tracked destination sweeps it into the wrapper's
publishing commit. Never delete either: the leftover tdata/working is what blinds the next
run (the app shows its "previous launch was not finished properly" window instead of starting,
so the run writes no test_log.txt and reads as a hang), while a leftover .dmp never blocks
a launch and is moved only to keep a later run's dumps report free of old minidumps.
test-run names every moved file and its destination in stale_crash_cleared, refuses to
launch when the report itself cannot be moved, and leaves a minidump it cannot move in place,
reported with a null destination.TelegramForcePortable exists without the marker, it is the user's real data: move it to
real_TelegramForcePortable when that is absent. If real_... already exists, the unmarked
live folder is the user's manual restore of that same preserved data — recursively delete the
live folder only.test_TelegramForcePortable to TelegramForcePortable, then create the marker file
TelegramForcePortable/testing (any content, e.g. echo 1). Never rename, modify, or delete
the golden folder.Any live/real folder combination is never a blocker. After SETUP the live folder is a marked test
copy, the golden folder is untouched, and real_... may or may not exist.
NO CLEANUP — the flow performs no folder operations after testing, ever. The marked test copy
stays live, and the three folders are never copied, moved, or deleted between testing phases. SETUP
may move stale crash files from inside the marked live copy before a launch; this is not a folder
operation. The flow never restores real data to live: when the user wants manual use they copy
real_... to TelegramForcePortable themselves (keeping real_... in place), and the next SETUP
handles that unmarked live folder by step 3.
Deletion guard — the only folder the flow may ever delete is a live TelegramForcePortable that
either carries the testing marker or coexists with real_... (step 3). If the test account
breaks mid-loop (login screen, AUTH_KEY_DUPLICATED), delete the MARKED live folder (that deletion
takes the Crashpad database under tdata/dumps/completed/ with it, so copy out any dump worth
keeping first), re-run SETUP for a fresh golden copy, and retry once; if it is still broken the run
is UNRECOVERABLE. Never delete or alter test_... or real_... under any circumstances.
Serialize app runs. Never have two Telegram.exe instances alive against this account at once —
concurrent reuse of one auth key can trigger a server-side session reset. Before SETUP, launching, or
rebuilding, kill any straggler of THIS checkout's binary only — the one whose full executable
path is EXE (out/Debug/Telegram.exe in this checkout). Match on the full path; do NOT blanket-kill
every Telegram.exe on the machine. The user may be running a system-installed client or another
checkout's build against unrelated accounts — those use different auth keys, never conflict with this
account, and MUST be left alive. On Windows, scope the kill by path:
$exe = (Resolve-Path "$EXE").Path
Get-CimInstance Win32_Process -Filter "Name = 'Telegram.exe'" |
Where-Object { $_.ExecutablePath -eq $exe } |
ForEach-Object { Stop-Process -Id $_.ProcessId -Force }
taskkill /IM Telegram.exe /F is forbidden here and anywhere else in this loop — it is image-name-wide
and takes down the user's unrelated clients. Every "kill stragglers" / "taskkill" step below means
this path-scoped kill. The workspace helper's test-run and test-cleanup commands implement it
on every platform; prefer them over hand-written kill shell.
Avoid account-fatal calls; cloud data is otherwise fair game. The overlay must never trigger logout / session-termination / account-deletion, and must not wipe the account wholesale. Tests that genuinely need those use a separate burner account, not this one. (If a permanent destructive-call fuse is later added to the debug build, this is enforced in code; until then it is the evidence author's responsibility.) Everything short of that is allowed: this is a test-server account, so freely CREATE content in any chats (messages, drafts, tables, media) and freely DELETE or clear content that test runs created — including leftovers from previous runs and sessions (e.g. clear the self-chat rich compose cloud draft before a run instead of designing around accumulated junk; the live test copy is reused across runs and tasks, so local AND cloud state accumulate — reset whatever state the test depends on at the start of the run). Don't delete anything the user placed on the account by hand unless the task says so.
Before writing a command, script, probe, or overlay:
Instrument-specific requirements:
When the wrapper marks a task Visual: layout, "looks right" is not a vibe — it is a small
computation, and the test MEASURES it. The wrapper's design-spec phase writes the contract to
<WORK_DIR>/visual.md; impl builds to it; this loop verifies it. (Tasks marked Visual: appearance
use the ordinary visual/asset check above. Unmarked non-visual tasks use their applicable text or
behavior checks; only legacy unclassified visual changes use the visual/asset branch.)
Build the contract from the strongest available design evidence: explicit request relationships;
supplied references when present; current or legacy task-adjacent UI; then the closest established
desktop component/style token while preserving unspecified behavior. A mockup gives relationships,
never desktop pixels. With no mockup, repository anchors and the written requirement provide those
relationships. The strongest anchor is an existing widget: "the count badge IS the dialogs-list
unread badge" pins font + height + padding to st::dialogsUnread* and is self-correcting — far
better than "a blue circle ~24px". Cite each source and record every inference; never invent a
reference or arbitrary geometry merely to fill the contract.
Write it as an ORDERED DERIVATION: each step resolves one quantity the next consumes, so impl and test are both mechanical. Example — a glyph-on-rounded-square icon + title + count, in a bubble:
Anchor: T = st::<title>.font->height ; Badge := the dialogs unread-badge metrics
1. glyphH = 1.4·T ±2px — white glyph box height (from T)
2. square = glyphH ÷ (2/3) ±2px — accent rounded-square side ; iconR = square·0.28
3. margin m (equal on square's top/left/bottom) ; bubbleH = square + 2·m ±1px
bubbleR = bubbleH/2 ; iconR : bubbleR must read as in-sync (icon proportionally smaller)
4. titleY = (bubbleH − T)/2 ±1px — title vertically centered in the bubble
5. badge = Badge (font+height+padding) ; vertically centered ; margins top=right=bottom equal ±1px
Then the RELATIONSHIP checks that catch what existence-checks miss — each falsifiable: square ≤ bubbleH (no overflow/overlap), the square's three margins equal, the two corner radii in sync, the
badge identical to a real chat-row unread badge. Note each source-to-desktop adjustment and which
token or metric grounds it; describe mobile→desktop conversion only when a mobile reference exists.
How TEST verifies it (numbers over eyes):
font->height and the
QRect of each piece (glyph, square, bubble, title, badge) — and assert each derivation line
arithmetically within tolerance. Live-widget geometry is the primary oracle; it deterministically
catches "icon taller than the bubble", "square overflows", "badge oversized / cramped". Where a
rect can't be logged, measure it from a tight crop by colour (accent square, badge, bubble outline
are separable).Visual: layout check APPROVES only when the measured geometry satisfies the contract; any line
out of tolerance is an IMPL_BUG (report measured-vs-target) and loops like any other.The repository carries a permanent test harness under
Telegram/SourceFiles/test/ — always compiled, runtime-gated on -testagent
(Test::Active()), with all of its #ifdefs inside the harness itself:
Read Telegram/SourceFiles/test/README.md completely before designing,
authoring, or recovering an overlay. It is the decision guide for stage
semantics, exact-object publication, input targeting, capture selection, the
specialized helpers, and first-run diagnostics. Then read the headers for the
helpers selected by the design. Search the directory before writing local
scaffolding.
test_runner.h — the staged scenario engine: Stage{name, run, until, then, timeout},
waitEvent, waitForSessionReady, the normal bounded non-fatal waitForChatsLoaded(), and
explicit strict waitForChatsLoadedStrict(); timing out an ordinary Stage ends the whole
scenario, while the wall-clock watchdog (default 120s, TDESKTOP_TEST_WATCHDOG override)
guarantees TEST_COMPLETE + quit on every exit path including timeout. actOnWidget waits for
and lifetime-guards the exact target before acting once. captureAndInspect saves the accepted
prepared frame and then runs numeric/raster assertions against that same widget and image.test_log.h — evidence dir from TDESKTOP_TEST_EVIDENCE_DIR (the workspace test-run
helper sets it), flushed absolute-path logging, Step/Pass/Fail/Check/Note, CheckNear
tolerance assertions, LogGeometry, the standard markers.test_widgets.h — FindAll<T>/FindFirst<T>/FindVisible<T> (the dynamic_cast-based
finders that avoid the guaranteed findChildren<CustomWidget*> crash), Click, TypeText,
CommitText, PressKey, Drag, and Wheel via real Qt events, plus generation-counted,
lifetime-guarded PublishLiveWidget / PublishLiveAction seams for exact layer-owned objects;
each delivered event runs with postponed-call processing
deferred and is followed by a drain of every pending Ui::PostponeCall to empty, so
postponed input fix-ups AND their own change handling are settled when the helper returns;
wrap programmatic setText in Test::Settle.test_capture.h — CaptureWidget/CaptureRect (visibility check, QWidget::grab() so
floating elements and locked desktops cannot occlude, automatic blank-image FAIL, geometry
log, SCREENSHOT marker), plus PreparedWidgetCapture; Runner::captureWidget polls an
exact target until it has a valid painted frame and saves that same accepted frame,
Crop/Zoom/ContactSheet for tight same-scale evidence.test_ink.h, test_style.h, and test_panel.h — painted-ink/contrast measurement, settled
palette baselines, and detection of a Ui::SeparatePanel show-animation cache.test_messages.h, test_transfer.h, test_open_handoff.h, and test_launch_fuse.h — direct
sent-message, document-transfer, document-open, and safely blocked OS-launch oracles. Prefer
these established task-specific observers over reconstructing the same state in a scenario.test_agent.h — Test::Fire(name) / HasFired(name) named waitpoints;
launch_finished fires at the end of Application::run(). TDESKTOP_TEST_SCALE is applied
by the harness at startup.test_scenario.cpp — the overlay-owned slot: it defines Test::SetupScenario(runner) and
is a no-op in the repository.A minimal scenario shape:
void SetupScenario(not_null<Runner*> runner) {
runner->waitEvent(u"launch_finished"_q);
runner->actOnWidget(
u"activate the published target"_q,
[] { return Test::ReadLiveWidget(u"task.target"_q).widget; },
[](QWidget *widget) { Test::Click(widget); },
[](QWidget *widget) {
return widget->isVisible() && widget->isEnabled();
},
Test::kDefaultStageTimeout,
[](QWidget *widget) {
return u"visible=%1 enabled=%2 size=%3x%4"_q
.arg(widget->isVisible())
.arg(widget->isEnabled())
.arg(widget->width())
.arg(widget->height());
});
runner->captureAndInspect(
u"target_surface"_q,
[] { return Test::ReadLiveWidget(u"task.surface"_q).widget; },
[](QWidget *widget) {
return widget->property("contentGeneration").toInt() > 0;
},
[](QWidget *widget, const QImage &image) {
Test::LogGeometry(u"surface"_q, widget->geometry());
Test::Check(
image.width() >= widget->width(),
u"capture covers the surface width"_q,
u"image=%1 widget=%2"_q
.arg(image.width())
.arg(widget->width()));
});
}
The overlay starts by replacing test_scenario.cpp with the task's scenario. Author the
scenario fresh against the CURRENT implementation from the task's check design; never re-implement
logging, finding, capturing, watchdogs, or quit handling that the harness already provides —
re-derived scaffolding is where capture flaws come from. The scenario is the complete runtime
driver of first resort: prefer
programmatically triggering every required action and judging the saved logs and captures
afterwards over any external desktop driver, whether or not one is available. Drive the whole
task-specific flow inside the Debug binary on the event loop, waiting for observable state,
logging assertions, capturing the rendered target in-process, and quitting. A locked macOS
session does not reduce required coverage and is never a testing blocker. The scenario runs
only when -testagent was passed AND the live portable folder carries the testing marker,
so it can never run against real account data. The overlay must:
Prefer keeping the first run centralized in test_scenario.cpp, but never treat that module as
a sandbox boundary. After a setup or reachability failure, inject probes, fixtures, callbacks,
waitpoints, or direct test entry points at any relevant tracked production location, including
initialized submodules. Choose the location closest to the changed code that preserves a real
execution of that code. Scattered test injections are acceptable when they remove fixture
assumptions; all must remain disposable, inventoried, and excluded from implementation commits.
Keep any code added outside test/ inside #ifdef _DEBUG blocks only when it would
change release behavior; harness calls like Test::Fire are runtime no-ops and need no
guard.
Pick a test strategy and record it in the spec:
live-data (use real account data) · live-mutate (really create an entity — prefer a
throwaway target, clean up after) · inject (build fake local state without the network) ·
mock-api (intercept specific requests, return canned responses — for payments/destructive).
Prefer inject over live-mutate to avoid account/server accumulation and flake.
Express the flow as Runner stages with condition-waits over fixed timers (an until
predicate on the target widget/data actually existing, with the stage timeout as fallback).
Fixed sleeps are the main source of screenshot flake.
Keep the stage roles separate: .run is a one-shot action only after an earlier stage established
its prerequisites; .until is a pure readiness observation with no mutation and no expected
product result; .then performs assertions/actions; timeoutDetails logs the latest concrete
identities and values. If an action needs the object this stage is waiting for, use
Runner::actOnWidget. An expected mismatch in .until becomes a misleading timeout instead of
an implementation verdict.
Prefer PublishLiveWidget / PublishLiveAction at an inventoried production construction or
callback seam for repeated boxes, wrapper-owned buttons, and async replacements. Wait for the
published generation and invoke the exact action; do not rediscover those objects by descendant
order.
Log through test_log.h (Step/Pass/Fail/Check/Note/CheckNear/LogGeometry) —
it already writes the flushed absolute-path log and the exact TEST_STEP / TEST_RESULT /
SCREENSHOT / TEST_COMPLETE markers the external runner parses. Never hand-roll marker
strings or log files.
Capture the target tightly with CaptureWidget/CaptureRect — the specific widget /
row / glyph, unambiguously in frame at usable resolution. A full-window grab that leaves the
target clipped, off-screen, or thumbnail-sized is NOT acceptable evidence — if the target
isn't clearly captured, that is a TEST_FLAW (re-frame), never a pass. The helpers grab
in-process after layout and paint, so a locked desktop never blocks capture and a blank
grab fails loudly instead of passing silently.
For a full box, layer owner, animated root, or any surface whose children
appear asynchronously, use Runner::captureWidget. Resolve the exact
target on each poll and put task-specific content/identity checks in its
optional readiness predicate. It waits for a visible, non-empty, nonblank,
valid paint root and saves the exact frame that satisfied readiness. Do not
hand-roll GrabWidget + LooksBlank + SaveImage for evidence and do not
assume that object construction, isVisible(), or one child paint event
means the owning presentation has painted current content.
When the capture itself needs geometry, colour, luma, containment, or raster assertions, use
Runner::captureAndInspect. Its readiness predicate identifies current painted content only; its
inspector logs and asserts the result after saving the exact accepted frame. Do not put expected
height/colour/pixel values in capture readiness, where a product bug is misclassified as timeout.
Lay down the oracle's references. Save every applicable independent reference beside the
crop (SaveImage, ContactSheet for same-scale comparison). Exact asset work saves OLD and
intended-NEW art as <name>_{old,new}.png. Without target artwork, save the
baseline/reference-component crop when available and log the contract anchors,
style/resource identities, and measurements. Never fabricate an _new image.
Prefer asserting on logged state (log the actual value, assert on text — deterministic); reserve screenshots for genuinely visual checks where an eye is the right judge.
Rely on the Runner's built-in watchdog and termination: it force-quits at the wall-clock cap
and ends every path — success, assertion failure, or stage timeout — with TEST_COMPLETE
then quit, so the app never hangs holding a lock on the exe. Do not install a second
watchdog and do not call Core::Quit() from scenario code.
Telegram's custom widgets (Ui::InputField, Ui::FlatLabel, Ui::RpWidget, boxes, buttons, …)
do NOT declare Q_OBJECT — they have no own meta-object. So QObject::findChildren<T*>() does
not filter by type for them: with no distinct meta-object it matches the nearest moc'd base
(QWidget), i.e. it returns every child widget blindly cast to T*. The moment you use one as
T (e.g. call InputField::setFocused() / rawTextEdit() on what is really a VerticalLayout) you
get a raw SIGSEGV — the debugger shows this with the wrong dynamic type. A clean rebuild does NOT
fix it; it is a real bug in the overlay, not a stale build.
findChildren<Ui::SomeCustomWidget*>(). Use the harness finders
Test::FindAll<T> / Test::FindFirst<T> / Test::FindVisible<T> from test_widgets.h,
which enumerate findChildren<QWidget*>() (QWidget is Q_OBJECT, so that call is sound)
and dynamic_cast each result — C++ RTTI identifies the real type regardless of Q_OBJECT.Q_OBJECT types (QWidget, QLabel, QLineEdit, …) are safe to pass directly to
findChildren<T*>().FindVisible<T> filters only on isVisible(). It does not prove current-layer ownership,
non-empty mapped geometry, unobscured paint, hit-testability, or latest-instance identity. Add
those predicates or publish the exact live object; do not treat the helper name as a stronger
visibility contract.The Windows launcher changes the working directory to the exe folder before the app runs, so a
relative log path silently fails to write (QFile won't create missing parents) — the run
looks "clean" but produces no evidence. Test::EvidenceDir() resolves
TDESKTOP_TEST_EVIDENCE_DIR (which the workspace test-run helper always exports as an
absolute path) and creates it up front, so harness logging and captures are immune; never
bypass it with hand-built relative paths.
<WORK_DIR>/test-overlay.paths normally starts with
Telegram/SourceFiles/test/test_scenario.cpp and then lists every in-situ injection or
Test::Fire path. It may name any tracked file in the source checkout or an initialized
submodule; no unrelated or untracked path may be used. After building, save the overlay with the
workspace helper's overlay-save command: it verifies every dirty path against the inventory,
writes the top-level patch plus a per-submodule patch bundle when needed, and restores only the
inventoried overlay paths to their repository baselines; never hard-reset the repository. The
overlay never enters an impl or submodule commit.overlay-apply (a --3way
application that reports conflicted paths). This succeeds ~90% of the time when the tail change
was small.<WORK_DIR>/test.md (which
records injection point, fake values, and assertions) rather than fighting conflict markers.
Scenario steps that only call public APIs should live in their own block so they never conflict;
only true in-situ injections land inside impl files.This section applies only to selected Telegram runtime or overlay checks. Other instruments run their assessed command directly and retain the same exact command, environment, exit-code, log, artifact and control evidence.
EXE normally,
run the in-binary overlay flow, collect its logs and widget/window grabs, assess them, and clean up.
Do not try to unlock the session and do not return BLOCKED because the lock screen is present.BUILD. A single changed TU compiles fast; only the overlay-touched files + link
rebuild between rounds. On Windows, run the shared exact-path proactive cleanup before every
build. If the build reports LNK1104, C1041, access denied, or file in use, follow
.agents/shared/build-lock-recovery.md and retry within its bounded budget..svg, etc.) without touching a .style, an incremental build will NOT re-pack
it and the binary keeps the OLD asset. Before building such a task force regeneration — touch the
referencing .style (or clean the codegen output) — so the change actually ships. A render that
shows no difference from before is the symptom of skipping this.test-run command with EXE and EVIDENCE_DIR. One call
performs the SETUP steps (Test account), creates EVIDENCE_DIR, path-scope-kills stragglers,
then, for a reused marked-live account, moves a non-empty live tdata/working to
<EVIDENCE_DIR>/stale-crash/working and every live tdata/dumps/*.dmp to
<EVIDENCE_DIR>/stale-crash/dumps/ before launch. A zero-byte tdata/working is neither moved
nor reported. It launches EXE with -testagent -noupdate (so a shipped update can never
replace the binary under test mid-run) capturing stdout to
<EVIDENCE_DIR>/app_stdout.txt and stderr to <EVIDENCE_DIR>/app_stderr.txt (the flag prevents
modal crash hangs, and stderr captures assertion text), enforces a hard wall-clock deadline
from launch and a quiet-log watchdog while polling <EVIDENCE_DIR>/test_log.txt, detects
TEST_COMPLETE versus process death (crash) versus the caps elapsing (hang), kills any
straggler, and returns one JSON report with the parsed markers, stderr tail, fresh crash
diagnostics, crashpad_dumps_added, death_signals, and stale_crash_cleared.
TEST_COMPLETE alone is not success: when the process writes it and then dies, the verdict is
died-after-complete, not complete, on any of three independent signals — a non-zero
exit_code, a new .dmp in the live tdata/dumps/completed/ Crashpad database across the run,
or a fresh top-level tdata/dumps/*.dmp from a Breakpad build. crashpad_dumps_added is that
before/after delta, listed in full because test-run never clears completed/ between runs;
death_signals names which of "breakpad_dump", "crashpad_dump" and "exit_code" fired, and
is [] for a healthy run. stale_crash_cleared is an ordered list of {from, kind, to}
entries whose kind is "report" or "dump", and is [] when nothing was cleared. If the
stale report cannot be moved, test-run refuses before launch, prints the helper error on stderr,
exits non-zero, and emits no JSON. If a dump cannot be moved, test-run leaves it in place,
records "to": null (a null destination), and continues to launch. Then read each SCREENSHOT:
image and judge it, save the binary overlay patch, and restore only inventoried overlay paths
(overlay-save — the patch must be saved before that restore). The runner only gathers evidence;
ASSESS below stays the agent's own adversarial judgement.-testagent)A Debug build normally turns a failed std::vector bounds check, a bad iterator, an assert(), a
pure-virtual call, or abort() into a modal Abort / Retry / Ignore dialog. That dialog blocks
the process forever — the agent sees no TEST_COMPLETE, no process death, just a hang until the
watchdog cap, and learns nothing about the cause. -testagent removes those dialogs. With it
set, the binary:
abort() message box (no button to press, never hangs);<EVIDENCE_DIR>/app_stderr.txt, tagged [testagent];-testagent implies -debug).Do NOT key a pre-TEST_COMPLETE crash decision on exit code. Breakpad handles the crash and
the process usually exits 0 — exactly as tdesktop's own crash detection assumes. The reliable
crash signals before TEST_COMPLETE are: the process is gone WITHOUT a TEST_COMPLETE marker,
AND a fresh non-empty <workdir>/tdata/working exists. On macOS, a fresh matching system .ips
report is also sufficient when Telegram's reporter wrote nothing. After TEST_COMPLETE the
opposite holds: CrashReports::Finish() unlinks tdata/working during the clean shutdown that
precedes a teardown fault, so the only signals left are a non-zero exit_code, a new
tdata/dumps/completed/*.dmp on the macOS Crashpad build, and a new top-level tdata/dumps/*.dmp
on the Breakpad builds — test-run reads all three and reports died-after-complete. So always
pass -testagent, and on a crash gather diagnostics in this order before
deciding the verdict:
<EVIDENCE_DIR>/app_stderr.txt — the [testagent] assert: … line gives the failed expression and
file:line (e.g. vector(1931) : … vector subscript out of range). Usually enough to localize.<workdir>/tdata/working — the crash report the reporter wrote: the Assertion: /
CrtAssert: annotations, the failed file:line, and Caught signal … / minidump id. Plain text;
read it directly. <workdir> is the launch -workdir (in portable test runs,
out/Debug/TelegramForcePortable/).<workdir>/tdata/dumps/ — the minidump (full stack, needs symbols to read; note its path in
test.md, don't try to symbolize inline). Breakpad writes *.dmp at that top level; the macOS
Crashpad build keeps its database one directory below, in <workdir>/tdata/dumps/completed/, so
a top-level listing can be empty while a real dump exists. test-run reports this run's fresh
top-level dumps in dumps and its new completed/ entries in crashpad_dumps_added, so on a
Breakpad build the first is the field to read and the second is always [].~/Library/Logs/DiagnosticReports/Telegram-*.ips — when the preceding files are empty,
inspect reports created after the exact process launch and match the app UUID/start time. These
reports can contain a fully symbolicated stack even when Telegram's reporter wrote nothing.A crash is an IMPL_BUG (the implementation tripped an assertion / dereferenced out of range), not
a TEST_FLAW, unless the overlay itself is what reached out of bounds — quote the [testagent] line
and the tdata/working excerpt in test.md as evidence, and feed the expression + file:line to the
impl-fix agent as the Root cause / Fix hint. Only a crash with NO usable diagnostic after one retry
is UNRECOVERABLE.
On macOS, treat this exact repeated startup signature as a stale Xcode incremental build, not an implementation or overlay verdict:
Lang Info: Loaded cached, keys: ...;.ips report shows SIGABRT in
std::vector<unsigned char>::operator[] →
Lang::Instance::applyValue() → fillFromSerialized() →
Local::readLangPack();Before changing recovery strategies, preserve the current overlay and account,
stop only the exact-path app,
follow the portable-folder safety-copy procedure in AGENTS.md, then run one full Xcode Debug
clean followed by BUILD (the configured-tree clean is normally
cmake --build out --config Debug --target clean). Restore only portable folders missing after
the clean, never overwrite survivors, and retain the external backup through one successful
post-build launch. Rerun the same scenario once. Record the preceding runs as TEST_FLAW caused
by stale generated-language objects; do not spend an implementation attempt or re-author the
overlay. If the identical signature remains after the single clean rebuild, resume normal crash
classification and the directness ladder. Never loop clean rebuilds.
A run that never reaches TEST_COMPLETE and never dies is a hang. Two independent guards catch it:
-testagent force-enables the built-in DeadlockDetector — a
ping thread that, if the main/event loop stops responding (a genuine deadlock or an infinite loop
on the UI thread), raises Unexpected("Deadlock found!") from a side thread. That crashes through
the same reporter, so the frozen main-thread stack is captured in the minidump and the process
exits on its own (key on the tdata/working report, not the exit code) — same diagnostics path as
a crash above. No agent action needed beyond reading tdata/working / the dump. Detection is
within ~30–90s of the stall.Core::Quit(). For that the runner enforces a
hard wall-clock deadline (~90s) from launch and, when it elapses, does the path-scoped kill
regardless of output. No legitimate auto-test runs anywhere near a minute, so this cap is pure
backstop — but it is what guarantees the agent can never wedge forever.Classify by which guard tripped: a DeadlockDetector crash with a real main-thread stack in app code
is an IMPL_BUG; the external cap firing is almost always a TEST_FLAW (the overlay didn't
drive to TEST_COMPLETE/quit) — re-author the overlay — unless the captured stack/log shows the
implementation itself wedged, in which case it is an IMPL_BUG. Repeated external-cap kills enter
the directness ladder above; the same timeout signature alone never blocks the task.
The on-disk EXE (out/Debug/Telegram.exe) always contains the compiled overlay after a test run.
Restoring source does not rewrite the binary. When the loop reaches a TERMINAL verdict (APPROVED,
BLOCKED, UNRECOVERABLE, or attempt cap), after the final path-scoped kill and exact-path source
restore, delete the built EXE so no overlay-laden test binary is left for
the user to launch by mistake — the workspace helper's test-cleanup --exe EXE --delete-exe does
the final path-scoped kill and the deletion in one call.
A clean, feature-ready binary is one BUILD away on demand. (Delete only on terminal exit — between
attempts the next round rebuilds the overlay, so the binary is reused there.)
ASSESS decides APPROVED / TEST_FLAW / IMPL_BUG. Default to not approved; a check passes only on positive, specific evidence — in the captured pixels or the log — that the change is present AND correct.
visual.md geometry, named style/resource identities, or a
cited current/legacy analogue also qualify. Missing mockups alone never means the oracle is missing.<WORK_DIR>/test.md) — human-readable, append per attemptThe file the human opens to see how testing went. The evidence author writes checks before running;
ASSESS fills Actual / Result and the verdict. Create one ## Attempt per implementation commit and
append one ### Run per execution. A TEST_FLAW adds a Run under the same Attempt; an IMPL_BUG fix
starts the next Attempt. Never overwrite history.
# Test report — <project>/<letter>: <title>
## Attempt <n>
### Run <m> — instruments <reading|command|artifact|unit|probe|component|telegram-log|overlay|computer-use|screenshot|mixed> — verdict <APPROVED|TEST_FLAW|IMPL_BUG|UNRECOVERABLE>
- Evidence directory: <EVIDENCE_DIR>
#### Test 1 — <aspect of THIS change>
- Instrument: <selected direct instrument>
- Expected: <observable effect the change should produce>
- Oracle: <what would make this check FAIL>
- Oracle source: <task fact / visual.md line / repo analogue / supplied image / baseline>
- Control / negative: <known-present control, negative case, or reachability proof>
- Observed via: <exact command/artifact/value/crop/runtime surface>
- Actual: <what is literally visible / logged>
- Screenshots: <after.png and any real reference crops; none only for a non-visual check>
- Result: PASS | FAIL
#### Test 2 — ...
#### Verdict reasoning
<1-3 lines tying the checks to the verdict>
#### Root cause / Fix hint (only if IMPL_BUG — the impl-fix agent reads this)
#### Failure signature (one line, for recovery comparison)
#### Recovery plan (TEST_FLAW reruns only: prior proof, failed assumption,
forbidden technique, next directness strategy)
TASK: <TASK_ID>
STATUS: <DONE|BLOCKED>
VERDICT: <APPROVED|reason if blocked>
ATTEMPTS: <n>
TOUCHED: <repo paths or none>
DISCOVERED: <none|present in result.md|inline concise follow-ups when the wrapper has no result.md>
NOTES: <one or two lines, or none>
Detailed reasoning stays in external AI task artifacts. The chat reply is only this block.