Back to Dyad

Correct state machines MVP evaluation

plans/correct-state-machines-evaluation.md

1.10.018.9 KB
Original Source

Correct state machines MVP evaluation

Scope and reproducibility

The common baseline is merged PR6 commit 0be0cb40a (refactor: add correlated actor request settlement (#4143)). That is the last merged main commit before either pilot changed its domain. Image-generation PR8 and app-run PR7 were developed in the opposite order, but both final pilot diffs descend from this foundation and are present in the PR9 base, 9ffdcca6d.

Reproduce the handwritten glue measurement from the repository root:

sh
node scripts/measure-correct-state-machine-pilots.mjs \
  --baseline 0be0cb40a \
  --post HEAD

The script uses the TypeScript scanner, excludes blank and comment-only lines, deduplicates overlapping ranges within a category, and prints the exact commit-specific file/range inventory with its JSON result. Tests, generated code, comments, and shared framework implementation are excluded. Moving a domain wrapper into another production file is still counted.

The post-migration inventories are:

  • App-run: src/app_run/remote_manager.ts, src/app_run/operations.ts, src/ipc/services/app_run_actor_service.ts, and src/hooks/useRunApp.ts.
  • Image generation: src/hooks/useGenerateImage.ts, src/image_generation/hooks.ts, src/image_generation/request_scope.tsx, src/ipc/services/image_generation_service.ts, src/ipc/services/image_generation_actor_service.ts, src/ipc/services/image_generation_operation_service.ts, and src/ipc/services/image_generation_definition.ts.

Enforcement shipped

  • defineFrameworkCoveredRemoteMachine brands migrated definitions only when they provide either a native runtime remote-intent contract or the narrow completion-aware protocol-v1 declaration/operation pair. App-run and image generation use that constructor. Production registration accepts only that capability or the exact legacy compatibility capability for chat stream, GitHub operations, plan handoff, and version preview.
  • The semantic AST inventory separates framework internals, migrated safe adapters, unrelated queues, and unsafe compatibility. Definitions and production capabilities are exact symbol inventories. Implementation boundaries use exact file ownership plus boundary counts, so additions, deletions, and file moves fail without coupling enforcement to private function or class names.
  • Every unsafe compatibility group records machine, exact file, mechanism, expected boundary count, rationale, and removal owner. No app-run or image-generation entry is in the unsafe compatibility inventory.
  • The reusable framework-mechanism suite covers duplicate coalescing/replay, conflicting identity reuse, exactly-once terminal settlement, bounded replay with pinned unresolved work, tracked producer drain, destructive commit/release, stale generation/release, abort/reopen, and inspected framework-owned resources.
  • assertNoOwnedResources reports every declared resource class with owner, machine, key, and generation. Existing pilot tests additionally inspect operation registries, request scopes, and transport subscriptions at their domain terminal/disposal boundaries.
  • The 46 foundation review findings and all 25 final PR7/PR8 review threads have exact catalogs. Domain-owned entries name the focused test rather than relying on a generic suite title. Known negative invariants remain decision blockers instead of being counted as successful coverage.

Exact compatibility inventory

Unsafe production compatibility remains only in these unmigrated owners:

Machine/domainExact filesMechanismRemoval owner
chat streamsrc/chat_stream/definition.ts, src/chat_stream/remote_manager.ts, src/ipc/services/chat_actor_deletion_fence.tsprotocol-v1 event widening, raw dispatch, ref-counting, deletion fenceConditional follow-up A/C
app/chat creationsrc/ipc/services/app_chat_creation_fence.tscreation/deletion counterConditional follow-up C
GitHub operationssrc/github_ops/useGithubOps.ts, src/ipc/services/github_ops_definition.ts, src/ipc/services/github_ops_service.ts, src/ipc/services/github_ops_presentation_service.tsprotocol-v1 widening, raw dispatch, deletion/reset counters, route mapConditional follow-up A
version previewsrc/hooks/useVersionPreview.ts, src/version_preview/VersionPreviewProvider.tsx, src/ipc/services/version_preview_definition.ts, src/ipc/services/version_preview_service.ts, src/ipc/services/version_preview_presentation_service.ts, src/ipc/services/version_preview_window_interest.tsprotocol-v1 widening, raw dispatch, bespoke waiter, deletion/reset counters, route/interest mapsConditional follow-up A/B
plan handoffsrc/plan_handoff/definition.ts, src/plan_handoff/remote_manager.ts, src/ipc/services/plan_handoff_service.tsprotocol-v1 widening and raw dispatch/enqueueConditional follow-up B/C
user inputsrc/user_input/read_model.tsindependent subscription ownershipConditional follow-up C

The executable source of truth, including every exact compatibility owner and its expected mechanism count, is compatibilityBoundaryInventory in src/distributed_machines/boundary_inventory.test_support.ts.

Migrated adapters are separately pinned:

  • App-run: one captured command-output enqueue and one completion-aware actor-service enqueue.
  • Image generation: two completion-aware protocol-v1 request dispatches and two destructive actor-service enqueues.
  • Both pilot protocol-v1 event-codec casts are classified as migrated declaration boundaries, not unsafe ordinary-caller escapes.

Shared and domain conformance evidence

The shared primitive suite is src/distributed_machines/testing/framework_mechanism_conformance.test.ts. Admission/authorization, request settlement, lifecycle, late producer, and renderer races that require a full transport or domain state are covered by the named framework and pilot tests cataloged in:

  • src/distributed_machines/testing/foundation_finding_catalog.ts
  • src/distributed_machines/testing/pilot_finding_catalog.ts
  • src/distributed_machines/remote_transport.test.ts
  • src/distributed_machines/operation_registry.test.ts
  • src/distributed_machines/actor_host_admission_gate.test.ts
  • src/app_run/main_actor.test.ts
  • src/image_generation/main_actor.test.ts
  • src/distributed_machines/use_machine_mutation.test.tsx

The shared resource inventory includes prepared requests, admitted operations, pending receipts, waiters, subscriptions/leases, fences/continuations, tasks/timers, producer sinks, actors, terminal payloads, renderer listeners, and renderer request owners. The reusable harness reaches zero in each terminal/disposal scenario it owns using registry and gate inspectors.

This audit does not prove one aggregate zero-resource snapshot for every domain-specific terminal permutation. Existing focused tests prove the individual app-run and image resource owners, but the lack of a unified domain-level inspector remains an authoring/diagnostic gap. There is also no single reusable runtime driver that instantiates both domain façades; the pilot-specific cases remain exact focused tests. This missing cross-pilot adapter is a conformance blocker, not completed evidence.

Historical review coverage and remaining blockers

All exact finding mappings are executable inventory tests. Three applicable image-pilot findings remain negative invariants:

  1. The image collection has one global actor key. Deleting app A fences and drains jobs for unrelated app B. The focused test pins the singleton key; safe app-scoped behavior requires partitioned actor keys or first-class scoped gate generations.
  2. prepareAppDeletion() publishes disposed settlement and cancels provider work before the database delete commits. If the database deletion fails, the current fence can reopen, but the provider work and settlement cannot be restored. The focused regression documents this irreversible pre-commit boundary.
  3. Closing the initiating image-generation window drops presentation instead of preserving the previous single-window fallback. The focused test pins the current no-peer behavior; a compatible fallback needs an explicit presentation ownership policy.

None is hidden by a widened allowlist. They are framework-covered lifecycle or compatibility problems and block expansion.

At the pilot merge cutoffs, accepted HIGH/P1 findings were addressed and no validated HIGH/P1 thread remained open. PR9's six-finder adversarial local review and fix recheck validated 13 deduplicated findings (4 HIGH, 9 MEDIUM). The implementation fixes the production registration bypass, exact-inventory gaps, resource and historical-evidence assertions, and measurement defects; it removes the false cross-pilot conformance claim and records the missing reusable domain driver as a STOP blocker. No validated local-review HIGH/P1 defect remains unresolved. Trusted-author PR review is processed separately before handoff.

Glue measurement

Positive percentages mean fewer handwritten framework-category lines. Negative percentages mean growth.

App-run

CategoryPR6 baseline LOCFinal pilot/PR9 LOCReduction
Admission/mutation221467-111.3%
Subscription/ref-count550100.0%
Promise/waiter/settlement15911627.0%
Deletion/fence012new
Late-producer guards8390-8.4%
Total518685-32.2%

Raw actor dispatch in the ordinary renderer manager fell from three callsites to zero. The app-run hook continues to call its domain manager façade; that is not raw transport. One completion-aware main enqueue and one captured command-output enqueue remain behind framework/domain façades. Unsafe migrated escape hatches: 0.

Image generation

CategoryPR6 baseline LOCFinal pilot/PR9 LOCReduction
Admission/mutation49289-489.8%
Subscription/ref-count00n/a
Promise/waiter/settlement66278-321.2%
Deletion/fence24113-370.8%
Late-producer guards/effect correlation59101-71.2%
Total198781-294.4%

Ordinary hook/component raw dispatch fell from two callsites to zero. Two completion-aware protocol-v1 adapter dispatches and two destructive actor enqueues remain behind the image domain façade. Provider deletion/reset counters and initiator routing are explicitly migrated domain-owned boundaries, not ordinary-caller escape hatches. Unsafe migrated escape hatches: 0.

Both pilots miss the required 30% reduction by a wide margin. The measurement counts new domain-specific request adapters and operation-service wrappers even when they delegate to shared primitives; excluding those handwritten production lines would misstate authoring cost.

Runtime and type-check cost

Measured on the same macOS checkout and Node/npm environment, after one warm-up:

MeasurementCommandResult
Shared conformance wall time/usr/bin/time -p npm test -- src/distributed_machines/testing/framework_mechanism_conformance.test.ts0.86 s wall; 7 tests, 312 ms Vitest duration
Targeted pilot suitethe exact 18-file command below3.75 s wall; 360 tests, 3.28 s Vitest duration
PR6 npm run tsthree warm runs1.78 s median (1.78, 1.79, 1.78)
PR9 npm run tsthree warm runs1.71 s median (1.82, 1.70, 1.71)

There is no material type-check regression. The contract/conformance presubmit is far below two minutes.

Reproduce the targeted pilot suite:

sh
/usr/bin/time -p npm test -- \
  src/distributed_machines/boundary_inventory.test.ts \
  src/distributed_machines/testing/framework_mechanism_conformance.test.ts \
  src/distributed_machines/testing/pilot_finding_catalog.test.ts \
  src/distributed_machines/testing/machine_conformance.test.ts \
  src/distributed_machines/remote_transport.test.ts \
  src/distributed_machines/operation_registry.test.ts \
  src/distributed_machines/actor_host_admission_gate.test.ts \
  src/distributed_machines/actor_host.test.ts \
  src/distributed_machines/prepared_request.test.ts \
  src/distributed_machines/remote_client.test.ts \
  src/distributed_machines/use_machine_mutation.test.tsx \
  src/app_run/main_actor.test.ts \
  src/ipc/services/app_run_actor_service.test.ts \
  src/image_generation/main_actor.test.ts \
  src/ipc/services/image_generation_actor_service.test.ts \
  src/ipc/services/image_generation_operation_service.test.ts \
  src/ipc/services/image_generation_service.test.ts \
  src/hooks/useGenerateImage.test.tsx

Compatibility, storage, and rollback

  • Distributed-machine wire protocol remains v1. No envelope version changed.
  • Existing app-run and image-generation IPC endpoints and renderer public methods remain present. The operation outcome/wait paths are additive pilot compatibility paths.
  • No database schema or migration file changed.
  • Golden behavior remains covered by the existing pilot renderer, handler, and E2E tests from PR7/PR8; PR9 itself changes no UI.
  • App-run rollback boundary: remove the native remoteIntent/correlated operation outcome path and restore the protocol-v1 manager adapter at the domain composition root. No persistent data conversion is required.
  • Image rollback boundary: remove remoteOperation, the operation-wait IPC, and the completion-aware request adapter to restore the PR6 transport shape. No persistent data conversion is required.

Known gaps and exclusions

  • Image deletion is globally exclusive and pre-commit provider cancellation is non-reversible.
  • Image result presentation lacks the prior fallback after its initiating window closes.
  • Domain-level aggregate zero-resource diagnostics are incomplete even though individual registries, scopes, subscriptions, and services have focused assertions.
  • No reusable runtime conformance adapter drives both pilot domain façades; the current shared suite covers framework primitives and the exact pilot catalogs link to focused domain tests.
  • Settlement is in-process and bounded; there is no crash-safe or durable exactly-once claim.
  • The protocol-v1 image adapter still requires two internal raw dispatch calls.
  • Presentation ownership, durable checkpoints, chat/plan queues, GitHub, version preview, generated bindings, inspector UI, persistence, and another domain migration remain out of scope.

Review-churn measurement

Behavior-preserving pilot refactors do not prove future review-churn reduction. The 50% normalized target remains unproven until two comparable future new machine or lifecycle/protocol migration PRs exist.

For each qualifying PR, record:

  • accepted framework-category findings per 1,000 changed production LOC;
  • review-fix commits per 1,000 changed production LOC;
  • raw finding, fix-commit, and changed-production-LOC counts;
  • reviewer count, review protocol, and review-window duration; and
  • duplicate, invalid, late, domain-policy, and framework-category classifications.

Do not combine unlike PRs or infer a reduction when fewer than two qualifying future PRs exist.

Conditional work

All expansion work is blocked. Do not begin presentation routing, OperationRouteRegistry, remaining remote-intent migrations, durable checkpoint pilots, chat/plan ownership, GitHub/version-preview migrations, generated bindings, or host representation work. Narrow corrective work may reduce the measured domain adapters or repair the two image lifecycle blockers, but it must be re-measured before reconsidering expansion.

Recommendation

STOP: the framework did not reduce bespoke lifecycle work in either pilot and retains named framework-covered lifecycle gaps; do not expand it.