Migrating go.d Collectors From V1 To V2

Requirement language follows the root AGENTS.md definitions.

This guide is for migrating an existing go.d collector from framework V1 to framework V2. It is not the starting point for a new collector; use src/go/plugin/go.d/docs/how-to-write-a-collector.md for new work.

V1 collectors are public integrations. A migration MUST preserve their existing user-visible contracts unless the user explicitly approves a breaking change.

Core Rule

A V1-to-V2 migration is compatibility work first. The clean end state is a V2 collector that behaves like the old collector from the user's point of view, with the old public contracts preserved and the internal collection path moved to collectorapi.CollectorV2 and metrix.CollectorStore.

Do not combine a compatibility migration with new enrichment, new topology, new host scopes, config expansion, chart redesign, or framework changes unless that work is required for the migration itself. If the migration reveals useful new work, split it into a later batch.

Before Writing Code

Create a compatibility manifest before implementation. The manifest can live in the active TODO or SOW. It MUST cover:

1. Module identity.
   • collector directory;
   • module name;
   • chart-template context_namespace and any group context namespaces;
   • go.d.conf toggle;
   • stock job config path;
   • service-discovery rules, if any, including src/go/plugin/go.d/config/go.d/sd/ and src/go/plugin/go.d/discovery/sdext/ references.
2. Registration and lifecycle.
   • current collectorapi.Register entry;
   • Defaults;
   • Init, Check, Collect, and Cleanup behavior;
   • any Once, reconnect, cache, or retry behavior.
3. Config contract.
   • YAML and JSON keys;
   • defaults;
   • validation;
   • config_schema.json;
   • stock .conf;
   • DYNCFG behavior.
4. Metric/chart contract.
   • chart IDs;
   • contexts;
   • dimension IDs and names;
   • algorithms;
   • units;
   • title, family, type, priority, multiplier, divisor, hidden, and float flags;
   • labels;
   • chart variables (Vars) -- STOP if present; see Chart Variables before implementation;
   • dynamic chart/instance generation rules;
   • chart lifecycle and obsoletion timing.
5. Integration artifacts.
   • metadata.yaml;
   • taxonomy.yaml;
   • health alerts;
   • generated integration page and README symlink;
   • COLLECTORS.md / plugin README entries when affected;
   • service-discovery docs such as SERVICE-DISCOVERY.md, when affected;
   • secrets docs such as SECRETS.md, when affected.
6. Tests and fixtures.
   • existing tests to preserve;
   • missing contract tests to add before or during migration;
   • real fixture coverage.

What Must Change

A V2 migration MUST replace the V1 collection path:

• registration uses CreateV2;
• the collector implements collectorapi.CollectorV2;
• New() creates metrix.NewCollectorStore();
• the collector stores metrix.CollectorStore;
• Configuration() preserves existing config return behavior;
• VirtualNode() preserves existing vnode behavior when the V1 collector has one;
• MetricStore() returns the store;
• ChartTemplateYAML() returns embedded charts.yaml;
• if the compatibility manifest contains V1 chart Vars, the migration stops until the Chart Variables decision path is resolved;
• Collect(ctx) returns error and writes observations to metrix;
• the completed migration removes the V1 Collect() map[string]int64 output path and any runtime bridge from V1 maps into V2 metrix.
• the completed migration removes Charts() and runtime collectorapi.Charts mutation from the production collection path.

Use src/go/plugin/framework/collectorapi/collector.go as the source of truth for the interface.

Parity During Development

Temporary V1 logic can be useful while developing the migration. For example, tests can compare V1 map output against V2 metrix observations after mapping both sides through the V1 chart manifest and any V2 chart-identity tooling that exists for the migration.

Do not invent an unreviewed local snapshot format when the comparison needs compiled chart identity that the framework does not expose. If the migration needs reusable chart-identity or alert-variable comparison helpers, treat that as a framework/test-helper prerequisite and follow src/go/plugin/framework/docs/changing-framework-code.md before relying on it.

That parity bridge is a development tool only. It MUST NOT remain in the final migrated collector runtime path. A finished migration that runs as V1-to-bridge-to-V2 is not a clean end state.

Runtime path means any code reachable from Init(ctx), Check(ctx), Collect(ctx), or Cleanup(ctx) during normal execution. V1 collection logic MUST NOT remain reachable from that runtime path, regardless of function names or return types.

If parity helpers are useful long term, keep only _test.go helpers or fixtures under testdata/. Before finishing, audit imports and prove no non-test file imports the old V1 path or parity bridge.

From src/go, run an import audit for the migrated collector:

go list -deps -test=false ./plugin/go.d/collector/<collector>/... |
  rg 'pkg/stm|plugin/go\.d/pkg/oldmetrix'
rg -n 'Collect\(.*map\[string\]int64|map\[string\]int64|collectorapi\.Charts|func .*Charts\(' \
  plugin/go.d/collector/<collector> -g '*.go'

The dependency command MUST return no runtime dependencies on old V1-only helpers such as stm, oldmetrix, or any collector-local parity bridge package. The source grep MUST NOT find a remaining runtime V1 map output, runtime collectorapi.Charts mutation, or Charts() path. If the old path is hard to identify, delete the temporary bridge and build/test the collector; the final runtime must still compile without it.

What Must Stay Stable

Unless the user approves a breaking change, the migration MUST preserve:

• module name and job identity;
• config field names and defaults;
• chart contexts;
• chart IDs;
• dimension IDs and names;
• dimension algorithms;
• chart title, family, type, priority, units, multiplier, divisor, hidden, and float flags;
• chart variable semantics used by health alerts;
• health alert lookups;
• metadata metric descriptions and units;
• source metadata content that drives generated integration docs;
• taxonomy coverage and CI behavior;
• service-discovery behavior;
• vnode behavior;
• user-facing lifecycle behavior.

If an existing collector has an accidental bug or inconsistent artifact, record it separately. Fix it in the migration only when preserving the bug would make the V2 collector incorrect or untestable; otherwise split the fix into its own tracked batch in the active TODO/SOW with owner-approved disposition. Do not close a migration with vague deferred items.

Implementation Shape

Prefer the same file ownership as new V2 collectors:

collector.go          # registration, New, public lifecycle, MetricStore, ChartTemplateYAML
init.go               # Init helper methods when setup is non-trivial
config.go             # Config, defaults, validation
collect.go            # Collect orchestration
collect_<area>.go     # separate upstream operations when there are several
metrix.go             # typed instruments built once
write_metrics.go      # observations into metrix
charts.yaml           # V2 chart template
*_test.go             # focused, table-driven tests

Keep public lifecycle methods in collector.go. Helper methods can move into focused files when that makes ownership clearer.

Lifecycle Rules

• Init(ctx) MUST perform setup and validation only. It MUST NOT collect the full metric set just to initialize state.
• Check(ctx) SHOULD be a cheap probe when the upstream API supports one. If V1 used full collection for autodetection, preserve the user-visible result while making the V2 path as light as the source allows.
• Collect(ctx) MUST write observations to metrix and return an error only when the cycle should abort. Fail-soft partial collection must be deliberate, tested, and logged without per-cycle spam.
• Cleanup(ctx) MUST preserve existing cleanup behavior and release any V2 Function or client resources added by the migration.

Metrics And Charts

Use typed metrix instruments and charts.yaml.

• Prefer creating instruments once when the metric surface is stable. Some collectors intentionally build instruments in the collection path when the surface is dynamic; if you keep that pattern, record why it is still the clean V2 shape for that collector.
• Use StateSet for fixed one-active-state values.
• Use Counter.ObserveTotal() when the upstream value is a source counter.
• Put multipliers, divisors, hidden flags, float formatting, instances, and label_promotion in charts.yaml, not ad hoc runtime chart code.
• Preserve V1 dimension algorithms exactly unless the old algorithm was wrong and the user approves the change.

When adding labels during migration, verify they are bounded and do not alter chart identity unexpectedly. Labels that improve filtering are acceptable only when they do not break existing chart/dimension contracts.

Chart Variables

V1 collectorapi.Chart.Vars have no direct charts.yaml / charttpl support today. Some shipped health alerts depend on those variables.

If the V1 collector uses Vars, the migration MUST NOT silently drop them. Choose one of these paths before implementation:

• add clean framework support by following src/go/plugin/framework/docs/changing-framework-code.md;
• preserve the alert semantics through an approved equivalent design;
• get explicit user approval for a breaking alert change and update health, metadata, generated docs, and release notes accordingly.

Until one of those paths is approved, migrating a collector that uses chart variables is blocked. Known V1 go.d collectors using chart variables at the time of writing include postgres, cockroachdb, hdfs, puppet, scaleio, whoisquery, and zookeeper.

Obsoletion Timing

V1 collectors often obsolete dynamic charts immediately with MarkRemove() / MarkNotCreated(). V2 chart templates expire unseen chart instances through lifecycle.expire_after_cycles and related chartengine policy. Exact immediate V1 timing is not always reproducible in V2.

For every V1 dynamic chart, record the old obsoletion timing and choose the V2 lifecycle policy deliberately. If the timing changes, document the behavioral change and get approval when it affects alerts or user-visible chart lifetime.

Dynamic IDs And Contexts

V1 collectors often build chart IDs with fmt.Sprintf. V2 templates derive contexts from context_namespace, group context namespaces, and chart context leaves. V2 autogen also has engine.autogen.max_type_id_len behavior.

The migration MUST prove that generated chart IDs, contexts, and dimensions match the old public contract, or explicitly record and approve any difference.

Config Rules

Migrations MUST keep existing YAML and JSON field names. Do not rename config keys to match new code style.

Do not add public config options as part of a migration unless they are required to preserve existing behavior. A proposed config option MUST name the concrete operator decision it enables; "operators may want to tune it" is not enough. Internal tuning SHOULD use constants. New user choices belong in a later feature batch with schema, stock config, metadata, and docs updated together.

autodetection_retry, update_every, and vnode are job/runtime fields in many existing collectors. Preserve the migrated collector's current YAML/JSON behavior and keep config.go, config_schema.json, stock config, and metadata consistent. Do not copy another collector's schema treatment for these fields without checking current framework expectations.

Host Scopes, Functions, And Topology

Host scopes, Functions, and topology are product design choices, not automatic migration side effects.

• If the V1 collector already has vnode behavior, preserve it.
• If adding host scopes would be useful but is not required for compatibility, split it into a later product decision.
• If the collector exposes Functions, isolate Function code in a dedicated <name>func/ package behind a narrow Deps interface. That interface MUST NOT expose, return, or embed *Collector.
• New topology producers MUST use src/go/pkg/topology/v1 and validate against src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json.

Tests

Migration tests MUST prove compatibility and V2 behavior.

Current shared helpers can prove schema validation, template compilation, and fixture chart coverage. They do not prove full V1-to-V2 chart identity parity, and they do not prove alert variables unless the migration records the variable source explicitly.

The non-negotiable minimum is alert-observable parity. The migration MUST prove that health alert on: contexts, referenced dimensions, referenced variables, and dimension algorithms/units used by alerts still resolve after migration. This is provable today with chart-template checks plus manual alert-variable review.

Exhaustive compiled chart-identity parity is broader: title, family, type, priority, multiplier, divisor, hidden, float flags, label promotion, and lifecycle policy. If existing exported helpers cannot observe those fields, either add a framework collecttest helper under src/go/plugin/framework/docs/changing-framework-code.md, or record a manual comparison recipe with exact fields and evidence. Do not claim exhaustive chart identity parity without one of those paths.

At minimum:

• config YAML/JSON serialization compatibility;
• Init, Check, Collect, and Cleanup lifecycle coverage;
• metric-store cycle behavior, including BeginCycle, successful commit, and abort on expected hard collection errors;
• alert-observable parity for chart contexts, dimension IDs/names, algorithms, units, and variables used by health alerts;
• chart template schema/decode/validate/compile coverage;
• chart coverage for fixture data expected to materialize all dimensions;
• health alert compatibility when alerts exist: each alert on: context must exist in the compiled template, and every variable referenced by alert calculations must still be provided by a dimension, variable-equivalent design, or approved alert change;
• generated integration artifact consistency when metadata/taxonomy changes;
• taxonomy coverage checks when chart contexts change;
• host-scope tests if scopes/vnodes are preserved or introduced.

collecttest.AssertChartCoverage is not a replacement for chart-identity parity. It verifies that emitted series and the template agree; it can still pass when both writer and template were renamed consistently.

Exhaustive chart-identity parity is required before claiming full compatibility. If the migration lacks a shared helper or recorded manual comparison recipe for the compiled fields listed above, claim only the narrower compatibility that was actually proven.

Until a shared alert-variable helper exists, use a manual grep/review pass for alert variables:

rg -n "\\$[A-Za-z_][A-Za-z0-9_]*" src/health/health.d/<collector>.conf
rg -n "Vars:" src/go/plugin/go.d/collector/<collector> -g '*.go'

Every referenced variable must still be supplied by a dimension, variable-equivalent design, or approved alert change.

Prefer table-driven tests using map[string]struct{} keyed by case name when cases share setup and assertion shape.

Parity manifest recipe

When chart-identity parity needs the compiled fields above and no shared helper observes them, this is a concrete form of that "manual comparison recipe": prove context/dimension/value parity by rendering BOTH collectors into one manifest shape, without depending on chart-ID string equality.

1. Freeze the V1 output as a golden manifest. For representative fixtures, run the V1 Collect() + Charts() and serialize, per chart: context, type, family, units, priority, and per dimension the name, algorithm, and de-scaled value (the V1 map[string]int64 value divided by the dimension Div). Commit the goldens under testdata/.
2. Render the V2 path into the same shape. Load the collector's LIVE ChartTemplateYAML() into a real chartengine, run a cycle through the real Collect() and metrix store, plan against the store, and read the plan's create/update actions into the same per-chart structure.
3. Compare structurally with a float tolerance. V1 truncates value*Div to int64 while V2 stores the true float, so compare values within a tolerance (for example 1e-3), not for exact equality. Assert that context, family, units, dimension names, and dimension algorithms match.

Compare chart IDs too when the migration preserves them (the default). Omit only chart-ID equality when the user approved a breaking chart-ID change -- then the preserved contract is the context, and chart-ID-keyed alert examples in src/health/REFERENCE.md must be updated. A gap (NaN or absent value) MUST fail the comparison loudly; never silently map it to 0.

The render MUST go through the collector's LIVE ChartTemplateYAML() and store, not a separately rebuilt template -- a test that rebuilds the template can pass even when the shipped ChartTemplateYAML() is wrong.

Validation

Run the narrowest commands that prove the changed contract. Typical migration validation includes:

cd src/go
go test -count=1 ./plugin/go.d/collector/<name>/...
go test -race -count=1 ./plugin/go.d/collector/<name>/...
timeout 15s go run ./cmd/godplugin -m <name> -d

For the load-verification command, success means the module is registered, a job starts, and the command keeps running until the timeout stops it. Treat unknown module, no jobs started, config-load errors, or an immediate exit before the timeout as failures. Use -c <config-dir> when the migrated test config lives outside the normal go.d config search path.

Also run framework or integration checks when the migration touches those contracts:

• chart template/framework changes: go test -count=1 ./plugin/framework/charttpl ./plugin/framework/chartengine
• metrix changes: go test -count=1 ./pkg/metrix/...
• runtime/framework changes: follow src/go/plugin/framework/docs/changing-framework-code.md
• metadata/taxonomy/generated docs: follow .agents/skills/integrations-lifecycle/consistency.md

Record exactly what ran. Full validation MUST NOT be claimed from a narrow command.

Commit Shape

Prefer small coherent commits:

1. Add missing compatibility tests, if needed.
2. Move registration and collection path to V2.
3. Convert charts to charts.yaml while preserving chart identity.
4. Update synchronized integration artifacts.
5. Add follow-up enrichment only in a separate batch.

If the migration requires a framework change, stop and follow src/go/plugin/framework/docs/changing-framework-code.md before implementing collector-local glue.

Anti-Patterns

• Rewriting chart IDs, contexts, or dimensions only because the V2 template makes a new name easier.
• Adding labels, host scopes, topology, or Functions in the same commit as the compatibility migration without a product decision.
• Shipping a V1-to-bridge-to-V2 runtime path after the V2 store is in place.
• Hiding framework gaps in collector-local helpers.
• Treating generated integration pages or README symlinks as authoring sources.
• Claiming compatibility without a manifest and tests.