v3/docs/adr/ADR-322-metaharness-flywheel-integration.md
@metaharness/{flywheel,darwin} as pluggable engines behind ruflo's ADR-176 self-improvement flywheelagent-harness-generator/packages/darwin-mode/package.json (@metaharness/[email protected]), npm @metaharness/[email protected] package surface, agentic-flow/docs/adr/ADR-075, and ruflo's own src/services/harness-flywheel*.ts / evolve-proof.ts / harness-improvement-ledger.ts.Phases 0–2 are implemented in the CLI package:
evaluateFlywheelCandidate is non-mutating. runFlywheelTick preserves implicit application for one compatibility cycle only when RUFLO_FLYWHEEL_LEGACY_APPLY=1.flywheel-receipt.ts implements canonical JSON, distinct candidate/run/receipt identities, deterministic paired-bootstrap decisions, Ed25519 domain separation, and independent statistical recomputation.flywheel-transaction.ts implements an O_EXCL cross-process lock and one atomic, directory-fsynced compare-and-swap state containing receipt consumption, ledger head, active champion, and serving epoch. Runtime materialization is derived and recoverable.flywheel-proposer.ts implements local|auto|darwin, fail-closed explicit Darwin selection, evaluation-only auto substitution, hard admissibility ceilings, and Pareto filtering before evaluation. The runtime consumes the resulting archive through candidatePolicies; Darwin adapters remain candidate generators and never gain promotion authority.ruflo metaharness evolve, bench, and flywheel run|status|receipts|history|promote are first-class CLI verbs. metaharness_flywheel provides the corresponding MCP surface.@metaharness/darwin@^0.8.0 and @metaharness/flywheel@^0.1.7 are optional dependencies guarded by scripts/check-metaharness-pins.mjs. Removing them leaves the local receipt, transaction, and proposer path operational.Phases 3–4 remain explicitly out of scope for activation: tool/model-policy evolution and unattended /loop promotion stay disabled until their separate privilege, spend, corpus-isolation, and production rollout gates are accepted.
The user wants ruflo to "better integrate metaharness" using the three published npm packages: metaharness, @metaharness/darwin, and @metaharness/flywheel. Grounding the current state (not from memory) changes the shape of the answer.
| Package | Version | Role in the "freeze the model, evolve the harness" thesis |
|---|---|---|
metaharness | 0.4.1 | Umbrella / scaffolder CLI — DESCRIBE: score (5-dim readiness), genome, mint, mcp-scan, threat-model, provenance sign/verify. |
@metaharness/darwin | 0.8.0 | The evolve ENGINE — PROPOSE: mutate one of 7 policy surfaces (planner, contextBuilder, reviewer, retryPolicy, toolPolicy, memoryPolicy, scorePolicy), sandbox-score each, keep measured wins, build a Pareto archive. Bin metaharness-darwin; evolve / `bench create |
@metaharness/flywheel | 0.1.7 | The LOOP — GOVERN: "a verifiable self-improvement loop … promote only what proves lift." Keywords: promotion, lineage, receipts. ESM library with . and ./cli exports (no bin — imported//cli, not npx). |
Ruflo did not wait for a published flywheel. It already ships a mature self-improvement loop in src/services/:
harness-flywheel.ts — runFlywheelTick: HARVEST a benchmark corpus from the install's real store (self-supervised) blended with the human-labeled ADR-081 anchor → BASELINE = active champion → PROPOSE neighbor configs and pick the best on a TRAIN split (a local deterministic hill-climb) → GATE the winner on a HELD-OUT split with a conjunctive rule: held_out_improves AND redblue(anchor-no-regress) AND drift ≤ thr AND replay-deterministic AND receipt_coverage AND canary-no-worse. Current implementation note: an accepted result calls applyChampionParams inside runFlywheelTick; this function is not a dry-run boundary.harness-improvement-ledger.ts: best-effort bounded JSONL. Accepted champions chain baselineRef == previous championRef within the retained window, but rotation discards old entries and write failures are swallowed. It is useful operational telemetry, not yet a complete append-only audit history.harness-flywheel-generations.ts: stateful, compounding lineage — each daemon tick uses the persisted champion as the next baseline. The latest promoted champion is automatically served at the start of the following tick (a one-generation delay), not held indefinitely for manual promotion.evolve-proof.ts: a single-round proof-of-mechanism. Held-out scores and significance are recomputed, but the verifier currently trusts the recorded canary rollback rate because the raw canary slice is not embedded.harness-flywheel-runtime.ts: binds the loop to the live neural store; opt-in via RUFLO_HARNESS_LOOP; never throws.Ruflo's gate evaluates more local safety signals than the 0.1.x package surface: anchor no-regression, canary isolation, drift bound, replay-determinism, and receipt coverage. Those signals are valuable, but the current persistence and serve boundaries do not yet justify claims of complete external verifiability or manual-only serving. The deterministic local path is pure Node and costs $0.
“Integrated” currently means that a capability has an MCP tool or plugin wrapper that an operator can invoke. It does not mean that the ADR-176 runtime flywheel calls it automatically.
| Capability | Current exposure | Used by ADR-176 runtime flywheel today? | Change required by this ADR |
|---|---|---|---|
metaharness score / genome | CLI subcommands, plugin scripts, and metaharness_score / metaharness_genome MCP tools | No | Admit versioned descriptors as optional fitness metadata; they do not authorize promotion |
| MCP scan, threat model, mint, audit/drift/similarity | CLI subcommands and MCP/plugin surfaces | No | Keep as describe/audit inputs; bind any promotion-relevant output into the receipt |
Darwin evolve | metaharness_evolve, ruflo metaharness evolve, and evolve.mjs; shells out through the pinned _darwin.mjs wrapper | Through the ADR-322B proposer contract when a Darwin invoker is supplied | Keep Darwin restricted to PROPOSE; ruflo remains promotion authority |
Darwin bench create|verify | metaharness_bench, ruflo metaharness bench, and bench.mjs; a verified suite can be passed to on-demand evolve | As an explicit proposal/evaluation input, never an authority input | Complete signed corpus-role manifests before unattended use |
| Darwin security bench | metaharness_security_bench MCP tool and plugin script | No | Treat as an additional security signal or CI job, not as a substitute for structural authorization invariants |
| Darwin GEPA | ruflo metaharness gepa, metaharness_gepa, and gepa.mjs for genome load/validate/render/analyze; optimization remains behind metaharness_evolve | No | Permit GEPA-derived candidates through the same proposer and authoritative ruflo gate |
Metaharness learn | ruflo metaharness learn, metaharness_learn, and learn.mjs; dry-run by default and requires a checkout for real runs | No | Keep opt-in and subject model-backed runs to the same immutable cost envelope |
| Red/blue adversarial testing | ruflo metaharness redblue, metaharness_redblue, and plugin scripts | Only through existing local gate-specific logic, not as the upstream Flywheel loop | Version and bind the exact red/blue evidence used by a promotion |
@metaharness/router | Hard dependency and independently callable routing product | No; modelPolicy is not an evolvable flywheel surface | Add constrained modelPolicy only after the signed SafetyEnvelope exists |
@metaharness/flywheel | Optional dependency plus API pin guard; ruflo-native flywheel CLI/MCP surface | Interoperability package, not promotion authority | Add projection cross-check before external receipt exchange is enabled |
The former CLI and dependency exposure gaps are closed. External Flywheel receipt projection remains intentionally disabled until a projection can preserve ruflo's stronger gate and evidence semantics without information loss.
@metaharness/darwin provides a broader candidate-search mechanism — 7 typed policy surfaces, sandbox scoring, crossover/epistasis, MAP-Elites niches, and a Pareto archive. Its practical superiority over the local proposer must be established on ruflo workloads through controlled quality, cost, latency, and reproducibility comparisons. Ruflo already shells out to Darwin for GEPA (gepa.mjs) but not as the flywheel's proposer.lineage/receipts model exposed by @metaharness/flywheel.model/router is not an evolvable surface. Ruflo has @metaharness/router (hard dep, ADR-321) but the flywheel doesn't evolve routing policy — upstream ADR-145's "8th surface."score, genome, gepa, learn, and related commands, while evolve and bench are MCP/plugin-only. There is no flywheel verb or MCP tool exposing the loop that already exists internally.runFlywheelTick both decides and applies. A safe dry-run CLI requires a pure evaluation result that can later be promoted with an explicit, integrity-checked command.The five latest dream-cycle issues were reviewed against this decision. They contain useful candidate signals, but they do not expand ADR-322 into a general memory, security, ensemble, performance, or swarm redesign.
| Issue | Signal relevant to ADR-322 | Routing decision |
|---|---|---|
| #2803 — MemIR provenance-role collapse | Promotion evidence needs typed origin, authority, subject, producer, and transformation lineage; a flat string is insufficient | Add typed evidence provenance to RufloFlywheelReceiptV1. The proposed AgentDB-wide schema and plugin SDK migration require a separate ADR |
| #2792 — heterogeneous reasoning ensembles | A heterogeneous topology may be a strong planner/modelPolicy candidate | Treat it as one candidate phenotype; it receives no privileged promotion path and must beat the same sealed holdout under the same cost envelope |
| #2783 — ShareLock and ChannelGuard | Per-tool inspection is insufficient for composed tool descriptions, and individually safe agents do not guarantee safe channels | Composite-tool inspection and inter-agent channel controls are structural preconditions for evolving those surfaces, not optional benchmark terms |
| #2778 — agentic inference and mixture-of-agents performance | Quality must be evaluated jointly with tokens, latency, money, and, when measurable, energy | Record resource observations in receipts and compare candidates on a declared Pareto policy; do not describe higher quality as “free” without measured equal-resource evidence |
| #2768 — least-privilege delegation | Privilege selection is a demonstrated orchestration bottleneck | Reinforces the signed, non-expanding SafetyEnvelope; an optimizer may select or narrow pre-approved grants but cannot mint or widen them |
The research signals are hypotheses and candidate generators, not promotion evidence. An arXiv preprint is not classified as peer-reviewed merely because it is recent; every benchmark claim used by the flywheel must record publication status, exact artifact/version, corpus, model, configuration, pricing snapshot, and reproducible run receipt.
The dream branches currently contain multiple unrelated files named ADR-320, and the #2803 branch contains ADR-322-dream-cycle-memory-typed-provenance.md. This ADR already owns number 322 for Metaharness Flywheel integration. Before any dream branch is merged:
Do not rip out the ADR-176 loop to adopt a 0.1.x package. Instead, treat the three npm packages as pluggable engines and a shared interop format behind ruflo's existing gate, in three tracks. Ruflo's gate currently evaluates a broader set of promotion conditions than the upstream package surface (held-out + anchor + drift + replay + receipt + canary + shadow) and stays the authority on promotion throughout.
ADR-322 is the architectural umbrella. Acceptance of this decision does not imply that all three implementation contracts ship as one change. The implementation is divided into independently testable specifications, each with its own schema version, feature flag, rollout, and acceptance report:
RUFLO_FLYWHEEL_TRANSACTION_V1.auto|local|darwin selection, package compatibility, candidate/archive translation, sandbox and model budgets, fallback behavior, and proposer comparisons.RUFLO_FLYWHEEL_DARWIN_V1.RUFLO_FLYWHEEL_RECEIPT_V1.No specification may rely on another specification's unversioned internal representation. Their contracts meet only through versioned policy, evaluation, promotion, and receipt schemas.
The A/B/C suffixes identify subordinate normative specifications owned by ADR-322; they do not allocate three independent entries in the global ADR-number registry.
@metaharness/darwin as the flywheel's PROPOSE/evolve engineMake the proposer pluggable behind a stable interface:
RUFLO_FLYWHEEL_PROPOSER=darwin → the PROPOSE step invokes @metaharness/darwin evolve (via the existing plugins/ruflo-metaharness/scripts/_darwin.mjs subprocess pattern, --sandbox real|mock|agent, --selection pareto, --mutator deterministic|ruvllm) and returns its Pareto archive as candidate configs.RUFLO_FLYWHEEL_PROPOSER=local → the existing local hill-climb (byte-reproducible, $0, no network).RUFLO_FLYWHEEL_PROPOSER=auto (the default) → use Darwin when installed and compatible. If Darwin is unavailable, record the attempted proposer, failure classification, and substitution in the evaluation receipt, then allow the local proposer to produce evaluation-only candidates. A substituted proposer cannot promote unless an administrator-signed policy explicitly allows that exact substitution; the default substitution policy denies promotion.darwin proposer fails closed if Darwin is unavailable or incompatible; it must not silently run a different optimizer.ruvllm mutator requires --confirm plus explicit immutable limits for maximum USD, requests, tokens, concurrency, and wall time. The implementation must enforce those limits locally and, where supported, at the provider; forwarding an opaque --risk-budget alone does not establish a monetary hard cap.The candidates Darwin proposes still pass ruflo's held-out gate before a promotion receipt can be issued — Darwin proposes, ruflo disposes.
@metaharness/flywheel as the interop lineage/receipt format + the standard surfaceRufloFlywheelReceiptV1 envelope and a canonical adapter to @metaharness/flywheel lineage commits. The adapter serializes numeric and structured policy values canonically; records the adapter, gate, corpus, and policy-schema versions; and carries the raw evidence needed to recompute every ruflo gate term.evaluateFlywheelCandidate(input) -> EvaluationPlan | EvaluationReceipt: pure with respect to the active policy; it may write a candidate receipt but cannot apply or serve it.promoteFlywheelCandidate(receiptRef, confirmation) -> PromotionReceipt: revalidates receipt integrity, gate version, policy version, corpus identity, expiry, and the current baseline before applying exactly once.evolve and bench to the main ruflo metaharness CLI map for parity with their existing MCP tools. Add a metaharness_flywheel MCP tool + ruflo metaharness flywheel <run|status|promote|history|receipts> CLI subcommand over the new evaluation/promotion interfaces. run is dry-run by construction. promote requires --confirm, refuses stale-baseline receipts, and is the only new command allowed to mutate the active policy. It must not call the side-effecting runFlywheelTick as its dry-run implementation.@metaharness/flywheel's reference rule as a projection cross-check in CI. Its Policy and PromotionEvidence types cannot represent all ruflo evidence directly, so CI must test the canonical projection separately from the authoritative ruflo gate. A disagreement fails the compatibility job and blocks claiming interoperability, but never promotes a candidate.All signed or hashed ADR-322C objects use the following normative format:
Canonical JSON: RFC 8785 JSON Canonicalization Scheme (JCS)
Digest: SHA-256
Signature: Ed25519 over domainPrefix || 0x00 || canonicalBytes
Content ID: sha256:<lowercase-hex>
Timestamp: RFC 3339 UTC, normalized to YYYY-MM-DDTHH:mm:ss.sssZ
NaN, positive/negative infinity, negative zero) are forbidden.Identity domains are separate:
candidateId = SHA-256(JCS(candidate policy))
evaluationRunId = UUIDv7 generated for each execution attempt
receiptId = SHA-256(JCS(unsigned receipt payload))
lineageId = UUIDv7 created once for a persistent evolutionary lineage
Re-evaluating the same candidate produces the same candidateId, a new evaluationRunId, and a new receiptId. Retries, independent verification, and multiple lineages therefore cannot be conflated with policy identity.
“Held-out improves” is not a raw positive delta. Every receipt embeds the complete, versioned decision rule and per-item paired observations. The initial default rule is:
relativeLift >= 0.02
AND pairedBootstrapProbability(candidate > baseline) >= 0.95
AND pairedBootstrapDeltaCILow95 > 0
AND frozenAnchorRegression <= 0
Domains may configure stricter thresholds or an approved alternative test, but cannot omit a noise/significance rule. A change to the statistical rule creates a new gate version and cannot be applied retroactively to an existing evaluation receipt.
For the default rule:
relativeLift = (candidateMean - baselineMean) / max(abs(baselineMean), metricEpsilon), where metricEpsilon is fixed by the versioned metric schema.SHA-256("ruflo/bootstrap/v1" || corpusHash || candidateId || baselineRef || evaluationRunId).pairedBootstrapProbability is the fraction of resampled mean deltas strictly greater than zero; pairedBootstrapDeltaCILow95 is the 2.5th percentile using the gate-versioned quantile rule.metaharness + darwin bench as the DESCRIBE/fitness backendmetaharness score/genome (already integrated) as the fitness descriptors and baseline readiness signal feeding the flywheel's selection.darwin bench create|verify, taskHash-stamped so it cannot be silently hand-edited), subject to strict corpus-role isolation:
Darwin's abstract policy surfaces map onto ruflo's real, already-configurable knobs:
| Darwin surface | Ruflo phenotype it mutates |
|---|---|
planner | swarm topology + strategy (hierarchical/mesh, maxAgents, specialized) |
contextBuilder | memory retrieval: HNSW ef/threshold, hybrid vs semantic, RRF/MMR weights |
reviewer | verification-quality gate thresholds (truth-score, adversarial verify depth) |
retryPolicy | hooks retry/escalation policy (3-tier routing thresholds) |
toolPolicy | Selection within an administrator-signed MCP tool/AuthScope envelope; evolution may narrow or choose among pre-approved bundles but cannot add tools, claims, secret scopes, network destinations, or privilege |
memoryPolicy | AgentDB tiers, consolidation cadence, EWC++ retention |
scorePolicy | intelligence-pipeline JUDGE verdict weights |
modelPolicy (8th, ADR-145) | Selection among administrator-approved @metaharness/router candidates and quality thresholds; immutable provider/model allowlists and per-run/per-day cost ceilings remain outside the genome |
The evolvable genome is subordinate to an administrator-signed SafetyEnvelope. Before sandbox execution and again before promotion, ruflo rejects any candidate that expands authorization, secret access, network reachability, model/provider access, concurrency, or spending beyond that envelope. Candidates that alter tool composition or inter-agent message policy must also clear the applicable composite-tool and channel guardrails before execution. These are structural preconditions, not benchmark scores and not overrideable warnings.
Resource constraints are also eligibility gates, not merely Pareto dimensions. Before ranking, a candidate must satisfy every hard limit in the signed envelope, including:
p95LatencyMicros <= envelope.maxP95LatencyMicros
costMicrosPerTask <= envelope.maxCostMicrosPerTask
tokensPerTask <= envelope.maxTokensPerTask
failureRate <= envelope.maxFailureRate
evaluationCostMicros <= envelope.maxEvaluationCostMicros
Only admissible candidates enter Pareto selection. Missing resource evidence fails eligibility unless the envelope explicitly marks that metric unavailable and non-authorizing. A candidate cannot compensate for violating a hard resource or safety limit by improving quality.
@metaharness/darwin and @metaharness/flywheel will be declared in optionalDependencies (^0.8.0 / ^0.1.7), not dependencies. Unlike @metaharness/router (a hot-path import, hard dep per ADR-321) and metaharness (hard dep), the flywheel and evolve engines are opt-in, heavy, explicitly-invoked loops. The ADR-150 removability constraint (“works without them”) stays in force. Graceful fallback applies only to auto; explicit engine selection fails closed.scripts/check-metaharness-pins.mjs + metaharness-pin-drift.yml) and verify package API compatibility, not merely semver freshness. Until those artifacts and optional dependencies land, this ADR records intent rather than shipped state.ruflo/flywheel-receipt/v1) so a witness signature cannot be replayed as a promotion signature.ADR-322A defines promotion as an atomic compare-and-swap transaction, not a sequence of best-effort file writes. The transaction is eligible only when:
activeChampionRef == receipt.baselineRef
AND receipt.decision == "accepted"
AND receiptState.promotedAt == null
AND ledgerHead == receipt.expectedLedgerHead
AND receipt.gateVersion == activeGateVersion
AND receipt.policySchemaVersion == activePolicySchemaVersion
AND receipt.safetyEnvelopeRef == activeSafetyEnvelopeRef
AND receipt.expiresAt > transactionTime
Within one serializable transaction, the promotion authority must:
baselineRef to candidateId.promotedAt, transaction ID, proposer identity/substitution status, and recovery metadata.The transaction commits all five state changes or none. Implementations may use a transaction-capable state store or a write-ahead transaction journal with deterministic recovery, but must expose one authoritative state machine. Policy materialization into runtime files occurs from the committed champion and serving epoch; it is idempotent and recoverable, never a second source of truth.
Concurrent attempts against the same receipt or baseline produce at most one successful commit. Repeating an already successful transaction returns its recorded result without reapplying the policy. A stale baseline, changed ledger head, consumed receipt, expired receipt, or changed safety envelope fails closed.
The implementation must trace controls and tests to at least these attackers and failures:
| Threat | Required control |
|---|---|
| Malicious candidate policy | Schema validation, signed non-expanding SafetyEnvelope, sandboxing, structural tool/channel checks, held-out and frozen-anchor gates |
| Compromised proposer process | Proposer has no promotion authority; canonical candidate output is treated as untrusted input; explicit proposer identity and substitution are recorded |
| Tampered benchmark corpus | Signed/content-addressed corpus manifest, sealed role assignment, task hashes, and corpus version bound into the receipt |
| Compromised evidence attestor | Attestor allowlist and scope, provenance per term, independent recomputation where possible, revocation and key rotation |
| Replay of an old accepted receipt | Baseline/ledger-head compare-and-swap, expiry, consumed state, lineage and safety-envelope binding |
| Concurrent promoters | Serializable transaction, unique constraint on receipt consumption and serving epoch, idempotent result |
| Compromised signing key | Purpose/domain separation, external key storage, rotation/revocation metadata, incident procedure that prevents new promotions and marks affected receipts |
| Evaluation resource exhaustion | Signed per-run and aggregate budgets, concurrency/wall-time limits, cancellation, sandbox quotas, and fail-closed budget exhaustion |
| Crash or partial persistence | Atomic transaction or write-ahead recovery; fault injection at every boundary; one authoritative champion after recovery |
Positive
modelPolicy closes the loop between the router product (ADR-148/149) and the evolutionary engine — "which model, when" becomes evolved, not hand-tuned.Negative / risks
@metaharness/flywheel's gate may differ; ruflo's is the authority. The package's gate is a CI cross-check, never the promoter.@metaharness/[email protected] will move fast and may break; the pin-drift guard + optionalDependency + graceful fallback bound the blast radius.Neutral
RUFLO_HARNESS_LOOP). At the time of this decision, the legacy generational runner auto-serves the previous promotion on the following tick; the new CLI/MCP surface must not inherit that behavior. Removing the two proposed optional engine packages leaves the local proposer and core CLI operational (ADR-150).RUFLO_FLYWHEEL_LEGACY_APPLY=1.RufloFlywheelReceiptV1, durable lineage, signature domain separation, independent verification, CLI parity, and CLI/MCP flywheel surfaces are present. run is non-serving; promote --confirm is explicit.auto|local|darwin, resource admissibility, Pareto filtering, and fail-closed/evaluation-only substitution semantics are enforced. Model-backed mutation remains disabled unless a caller supplies a separately budgeted Darwin invoker.SafetyEnvelope; then enable constrained toolPolicy and modelPolicy. Add role-isolated Darwin suites and the Flywheel projection cross-check in CI./loop only after fault-injection tests establish idempotency, crash recovery, complete lineage, no privilege expansion, budget enforcement, and explicit serving semantics. Evaluate convergence onto @metaharness/flywheel only once its evidence model is provably at least as strong as ruflo's.ADR-322 is not implementation-complete until automated evidence demonstrates all of the following:
SafetyEnvelope, including authorization, tools, claims, secrets, network, provider/model access, concurrency, or spending.auto proposer substitution is evaluation-only unless the active signed policy authorizes the exact substitution and the receipt records it.The load-bearing concurrency/fault test is:
Run 100 concurrent promotion attempts against one accepted receipt while injecting process termination at every persistence boundary. Exactly one promotion succeeds; one champion and serving epoch are active; one receipt is consumed; and the signed ledger verifies from head to genesis.
@metaharness/flywheel as the runtime, retire ADR-176. Rejected — a 0.1.7 package would replace ruflo's broader local gate (anchor/canary/drift/replay/receipt coverage) and its live-neural-store integration with something unproven. Convergence, not replacement.mapLimit/paretoFront primitives over a ruflo config genome, with evolve()'s source-surface mutation reserved for the harness files it actually owns.