code/lib/docgen-harness/README.md
A private test harness for the "docgen beyond React" work. It records what the legacy docgen pipelines produce today - extracted argTypes and generated code snippets - as reviewed snapshots, and holds the upcoming OSA engines to "current or better" against them. Nothing here ships to npm.
yarn test code/lib/docgen-harness # from the repo root
yarn test code/lib/docgen-harness -u # re-record after an intentional change, then review the diff
Each framework has three test files:
*-baselines.test.ts records argTypes and snippets per fixture and self-compares every committed baseline through the comparator.*-legacy-gaps.test.ts pins known legacy defects as test.fails red markers. They turn into hard requirements once baseline-path.ts flips from 'legacy' to 'osa'.*-render.test.ts smoke-mounts the fixtures.vue3 has a second recorder, vue3-component-meta-baselines.test.ts, because Vue ships two production docgen engines.
It drives the opt-in vue-component-meta path (docgen: 'vue-component-meta' in vue3-vite) over the same fixtures and writes cm--prefixed snapshots.
src/
├── index.ts # the comparator's public surface
├── compare/
│ ├── argtypes.ts # per-key argTypes rules
│ ├── snippets.ts # snippet rules + framework dispatch
│ ├── snippets-vue3.ts # Vue matcher
│ ├── snippets-angular.ts # Angular matcher
│ ├── parse-element.ts # root-element and attribute scanning
│ ├── parse-snapshot.ts # parser for committed argtypes*.snapshot text
│ ├── expect-current-or-better.ts
│ ├── is-snapshot-update-run.ts
│ └── types.ts
├── vue3/
│ ├── vue3-baselines.test.ts
│ ├── vue3-component-meta-baselines.test.ts
│ ├── vue3-legacy-gaps.test.ts
│ ├── vue3-render.test.ts
│ └── __testfixtures__/<case>/ # SFC, input.stories.ts, argtypes.snapshot, snippet-<story>.snapshot,
│ # cm-argtypes.snapshot, cm-snippet-<story>.snapshot
├── angular/
│ ├── angular-baselines.test.ts
│ ├── angular-legacy-gaps.test.ts
│ ├── angular-provider-seam.test.ts
│ ├── angular-render.test.ts
│ ├── compodoc-parsing-parity.test.ts
│ ├── csf-types.ts
│ └── __testfixtures__/<case>/ # component, stories, compodoc-input.json, aot-cmp.ts (signal cases),
│ # argtypes.snapshot, argtypes-filtered.snapshot, snippet-<story>.snapshot
├── svelte/ # planned
├── web-components/ # planned
└── perf/ # the performance bench, see below
├── PERF-METHODOLOGY.md # the measurement contract
├── docgen-perf/ # per-engine latency and memory suite, plus its engines/ and generators/
├── docgen-memory/ # the docgen-server memory regression gate
└── docgen-shared/ # sampling, stats, budgets and paths shared by both
expectCurrentOrBetter fails when a candidate loses anything a committed baseline records, and passes improvements.
other, where the legacy engine parked free text it could not resolve (TreeNode, Array([object Object]), { theme: string; dense: boolean }).
Such a stub accepts a candidate that adds populated structure (an empty enum/union/object is not an improvement) or resolves it to the scalar or single literal it already named; an unrelated scalar or literal is a lateral change and fails.
Only the three markers that record nothing at all accept any candidate: empty-enum, undefined, and the empty string - today's Angular and Vue spellings, so adding a framework means revisiting that list.
A resolution the rule cannot recognize (legacy TSFunctionType becoming a function sbType, say) fails rather than guessing; re-record and review the diff.
A recorded table.type.summary must survive (dropping it is a violation), but its text may change freely outside strictTable.
required, table.category, jsDocTags, control/action, and description/default contents are deliberately not compared (except required under strictTable); each would lock in a recorded lie (#28706) or engine-specific vocabulary.:x/v-bind:x, @x/v-on:x, #x/v-slot:x, and any .modifier all read as the same name.
The Angular comparison additionally gates root-element identity: the tag name must match and bare (valueless) attributes - the mangled attribute-selector markers - must survive.-u and reviewing the diff.-u run refuses to queue a regressed recording and stays red until the code is fixed.
Regressions fail with named violations; improvements pass.argtypes*.snapshot files are pretty-format text, not JSON.
parseArgTypesSnapshot reads them back, verifies itself by re-serializing every parse byte-for-byte, and rejects any parsed string carrying the writer-ambiguous entry-boundary shape.
A string whose unescaped write is byte-identical to real entry boundaries cannot be detected at parse time; the recorders' parsed-vs-live proofs guard that case on normal and CI runs, and on -u runs against the exact bytes queued for writing.Framework union and compilation fails at the switch in snippets.ts until the new matcher exists.The comparator machine-checks a deliberate subset: baseline arg names, description presence, default presence, table.type.summary presence, and type fidelity for argTypes; represented binding names, root-element identity, and bare-attribute survival for Angular snippets.
Everything else - description/default/summary text, table.category, control/action, per-arg jsDocTags, added args - is caught only by the byte-exact snapshot diffs reviewed at -u time, or by the sandbox gate's change findings.
Two flags scope trust to where the baseline earns it: legacyBaseline (only on legs whose baseline is a legacy compodoc recording) waives the raw false/NaN/null defaults that pipeline invents, and strictTable (only on the ACM self-ratchet, whose baseline the same engine recorded) additionally gates table.type.summary text changes and table.type.required true->false flips.
The sandbox baseline gate runs in the daily CI tier, so a whole-project regression can merge green and surface up to a day later, detached from the offending PR.
Known-accepted blind spots: enum members whose quoted and bare spellings collide normalize to the same member ('"small"' reads as small), and \r/\r\n in extracted strings are LF-normalized by vitest at write time, so a CR-bearing extraction can never record green (perma-loud, never silent).
vue3-component-meta-baselines.test.ts replicates the vue3-vite vite plugin's meta processing exactly - checker options, empty-meta skip, nested-schema pruning, exposed de-duplication, and the vue-docgen-api event-description backfill - so the cm- snapshots show what a vue-component-meta user actually gets today.
Keeping that copy in step with frameworks/vue3-vite/src/plugins/vue-component-meta.ts is manual; nothing detects drift.
cm- prefix keeps each recorder's stale-snippet guard scoped to its own files.sourceFiles records <sfc> because production stores the absolute module id and snapshots must stay path-free; nothing downstream reads it.schema: true) moved 17 of the 25 cm-argtypes.snapshot files and left every cm-snippet-* byte-identical.
The recorded state is whatever the lockfile resolves vue-component-meta to (3.3.9 today), so a dependency bump is a reviewed baseline change too.One directory per case; the recorders discover it automatically. The first run of a brand-new fixture fails once (snapshot files flush at suite end) - run it again and commit. Record thin or wrong legacy output as-is; never "improve" a fixture to make the legacy result look better. Snapshots must stay deterministic: no timestamps, no absolute paths.
input.stories.ts.<case>.component.ts (the class name must match the compodoc capture exactly) plus input.stories.ts and a captured compodoc-input.json.
Signal fixtures also commit an aot-cmp.ts with the ɵcmp input/output maps, captured once from real ngc output - JIT leaves them empty.
Stories import their CSF types from src/angular/csf-types.ts; the two runtime test files are excluded from the vue-tsc program because angular-vite client source is not strict-clean.Captures are pinned to @compodoc/[email protected].
Re-capturing with any other version is a reviewed baseline change - signal parsing drifts hard across versions.
Compodoc scans everything under the nearest package.json and ignores tsconfig include, so capture from a staging directory outside any Node package:
input.stories.ts) plus the case tsconfig.json into an empty directory, e.g. $(mktemp -d).npx -y @compodoc/[email protected] -p tsconfig.json -e json -d . there.documentation.json back as compodoc-input.json.cd code && yarn fmt:write.Nothing detects drift between a fixture's sources and its committed capture, so editing a component always means re-capturing in the same change.
table.jsDocTags stays undefined; component-level docblocks are not captured in script-setup SFCs.enum sbType, and the values keep their quote characters.convert() fallback (Array([object Object])).withDefaults() is extracted.defineModel('name') named models are invisible; snippets render a bare attribute instead of v-model:name.defineExpose members record no type at all - name and description only, where vue-component-meta resolves the same members to number and () => void.Number.MAX_SAFE_INTEGER lose precision in snippets.Pick-composed props record {}, recursive types a name-only stub, runtime array props type: undefined.defineProps<ReturnType<typeof useComposable>>() does not build in the legacy toolchain; a statement-block event expression crashes parse() outright (#23851). No baselines can exist for either.Fixtures reproducing open GitHub issues, to verify and close them when the OSA Vue engine lands.
Each has a red marker in vue3-legacy-gaps.test.ts.
cross-file-runtime-props/: imported runtime props must resolve to real argTypes (legacy records {}).cross-file-props-spread/: props spread from an imported call must be extracted.prop-slot-name-collision/: a prop must render as a prop attribute even when a slot shares its name.runtime-multi-constructor/: type: [String, Number] must become a structured union.runtime-proptype-cast/: literal unions behind PropType casts must keep their options.define-slots-literal-bindings/: defineSlots literal binding types must be extracted; the issue's own snippet repro is not covered here.slots/: scoped-slot binding types must be extracted; the marker covers only this symptom, not the issue's vue-component-meta repro.define-slots-with-props/: the issue's own repro - documented withDefaults props plus defineSlots losing all prop meta under vue-component-meta - does not occur at 3.3.9.
cm-argtypes.snapshot records descriptions, defaults, and slot docs fully intact; a regression baseline, no marker.
The issue's secondary HMR symptom is dev-server behavior outside this harness's reach.cross-file-union-alias/: imported literal-union aliases must unfold to their options.type-intersection-whole/: an intersection as the whole defineProps<> argument must resolve its props.model(), functions and undefined interpolate raw.required: true; compodoc never emits optional (#28706).NaN default; numeric expression defaults collapse to NaN too.Math.max(1, 3)).@deprecated vanishes (#9721), @see text leaks into the description, @default values keep quotes and a trailing newline.function, any, and generic type strings collapse to { name: 'other', value: 'empty-enum' }.angularFilterNonInputControls off, properties/methods/view child sections surface as argTypes, including private fields (#22007); on restricts to inputs.model() records one input plus a synthesized ${name}Change output; the compodoc quirk behind that is written up in code/lib/angular-compodoc/README.md.decorator-io-basics/: TS-optional inputs must record required: false. Red markers.jsdoc-tags/: member JSDoc tags must reach table.jsDocTags structurally. Red marker.decorator-union-enum/: the reported union collapse does not occur at compodoc 2.0.0; regression baseline, no marker.signal-io/: aliased signal inputs record under their alias at 2.0.0; regression baseline, no marker.properties-methods-noise/: the filter flag's origin case, and the fixture where both flag states meaningfully differ. The ACM engine closes it: propsTable: 'api' (its default) drops private and # properties and methods plus @internal members, while keeping protected members and every declared input and output, so the acm- baselines record fewer rows than the legacy ones on purpose.src/perf/ measures how fast the docgen engines are and how much memory they hold, which is the other half of the "docgen beyond React" question the snapshot comparator above answers for correctness.
It is a set of CLIs rather than part of this package's exported API, so nothing in src/perf/ is re-exported from src/index.ts.
All commands run from code/lib/docgen-harness:
yarn bench:docgen-perf # per-engine cold/warm latency and memory, full profile (~1 min)
yarn bench:docgen-perf --quick # smoke profile; its numbers are marked non-comparable
yarn bench:docgen-perf-gate # the same suite, plus budget assertions - what the CI gate runs
yarn bench:docgen-memory # the docgen-server memory regression gate
bench:docgen-perf generates synthetic projects under the shared sandbox directory, runs each engine in its own child process, and writes a results JSON next to them.
The generated trees are left on disk so you can open what was measured, and each engine/scenario owns one directory that the generator wipes before it writes.
That makes two bench runs at once clobber each other - one wipes a tree the other is mid-way through reading - so run them one at a time.
bench:docgen-memory asserts both that re-extraction is leak-free and that the program-recycle fix still flips a tight-heap run from OOM to survival.
yarn bench:docgen-perf --engine react-osa # one engine
yarn bench:docgen-perf --engine react-legacy --engine react-osa # a control pair, one invocation
yarn bench:docgen-perf --json /tmp/results.json # where the results land
The default run is react-legacy, react-osa, vue-docgen-api, vue-component-meta and compodoc.
Two ids sit outside it and only measure when named: react-legacy-rdt (the react-docgen-typescript parser) and vue-component-meta-next (the version-pair alias).
A ratio only appears when both sides of a control pair measured in the same invocation, so naming one side gives you a table row and no comparison.
Compodoc is skipped with a message when its CLI does not resolve; every other engine reads from the workspace, so a missing one is a failure rather than a skip.
The React engines run every scenario twice, because Storybook documents components in two shapes that cost very different things:
whole-index - one batch over every component, what the manifest generator does.first-story - the single component a request asks for, what the docgen server does. This is the number a developer waits for before Controls populate.The cold ratio between the React engines is 0.73 over the index and 0.08 over the first story; reading only one of them gives a misleading picture of the engine's cost.
vue-component-meta-next is an alias in this package's package.json, pinned to an exact version.
Point it at the version you want to test, yarn install, then run both sides in one invocation:
yarn bench:docgen-perf --engine vue-component-meta --engine vue-component-meta-next
Pin the candidate exactly rather than with a range - two caret ranges can resolve to one install, and then the run compares an engine against itself. The suite prints both resolved versions beside every ratio and calls out two equal ones as not being a comparison at all.
The mechanism is not Vue-specific: an engine entry declares which install it measures, the child imports that specifier instead of a hard-coded package, and a pair is two entries differing only in that field.
PERF-METHODOLOGY.md has the steps for setting one up on another engine, and the one case it does not cover.
PERF-METHODOLOGY.md walks through reading those guard lines, and through adding a pair for another engine.
Both gates run on CircleCI's daily tier, which is triggered on demand by the ci:daily label on a pull request - nothing schedules it, so this is not nightly protection.
bench:docgen-perf-gate runs the suite at the pinned profile, asserts the budgets in src/perf/docgen-shared/budgets.ts, and then proves its own failure detection by running a deliberately failing engine and requiring that run to come back non-zero.
It writes into the sandbox directory by default; CI passes --out ./perf-results so the results can be stored as a build artifact.
It refuses to report a green gate on a --quick run, on an empty budget table, or when a budgeted engine skipped or failed - each of those would look like protection while asserting nothing.
Budgets are ratios and absolute megabytes, never raw milliseconds, because wall clock on a shared CI executor is far too noisy to gate on.
Change one only against numbers measured on CI, and record where they came from in PERF-METHODOLOGY.md.
Read src/perf/PERF-METHODOLOGY.md before changing a metric, a budget, or a version pair.
It is the contract these numbers are only meaningful under.
The bench also carries unit tests for its own aggregation, reporting and generator logic, so yarn test code/lib/docgen-harness runs those alongside the fixture comparisons.