How collector taxonomy becomes integrations/taxonomy.json

Question answered: what is the general flow from collector metadata.yaml and taxonomy.yaml to the generated dashboard taxonomy artifact consumed by downstream frontend code?

Short version

metadata.yaml is the metric-context source of truth. Collector taxonomy.yaml files organize those contexts into the dashboard table of contents. integrations/gen_taxonomy.py validates both sides against the taxonomy registries and schemas, then emits the gitignored integrations/taxonomy.json cross-repo contract.

The implementation details can evolve, but the durable model is:

1. metadata declares what metric contexts exist;
2. taxonomy declares where those contexts belong and which widgets reference them;
3. the generator proves the references are valid;
4. the generated JSON carries the normalized section tree, placements, recursive items, and context snapshots.

Inputs

The taxonomy pipeline reads four source classes:

• Collector metadata.yaml files. The generator loads collector modules through the shared integrations loader and extracts metric contexts from metrics.scopes[].metrics[].name; see integrations/gen_taxonomy.py:269-276.
• Collector taxonomy.yaml files. These live next to collector metadata and use the closed v1 authoring schema integrations/schemas/taxonomy_collector.json.
• integrations/taxonomy/sections.yaml. This registry owns stable section_id targets and parentage for the generated TOC section tree; schema: integrations/schemas/taxonomy_sections.json.
• integrations/taxonomy/icons.yaml. This registry limits the icon IDs sections and placements may reference.

The field-level contract is documented in ../schema-reference.md. The contributor workflow is documented in ../recipes/add-go-collector.md and ../recipes/update-collector.md.

Metadata indexing

The generator first builds metadata indexes from all known collector metadata:

• by_path_module: matches a taxonomy.yaml file to its sibling metadata.yaml module by path, plugin_name, and module_name.
• all_contexts: sorted global list of known metric contexts, used for prefix resolution.
• contexts_by_plugin: contexts grouped by plugin name, used for collect_plugin selectors.
• dynamic selector guardrails from metrics.dynamic_context_prefixes and metrics.dynamic_collect_plugins.

The relevant implementation is integrations/gen_taxonomy.py:286-315.

This is why metadata.yaml is the metric source of truth: a literal context in taxonomy authoring is valid only if the sibling metadata module declares it. A taxonomy file can organize and reference metric contexts; it cannot invent static metric contexts.

Taxonomy authoring validation

Each collector taxonomy.yaml is loaded and validated against the closed authoring schema before semantic validation. The schema rejects old or ambiguous shapes such as placement-level contexts:, section_path:, and string shorthand in display-only positions.

After schema validation, the generator checks:

• section_id exists in sections.yaml;
• icon IDs exist in icons.yaml;
• literal owned contexts exist in the sibling metadata;
• literal widget references exist in metadata unless they carry the explicit unresolved escape hatch;
• dynamic selectors are declared by metadata guardrails;
• display widgets reference contexts but do not own them;
• every literal widget reference is owned somewhere else unless it is deliberately unresolved.

The matching and semantic validation start in integrations/gen_taxonomy.py:745-790. Selector and literal-reference validation live around integrations/gen_taxonomy.py:438-526.

Ownership model

The generated artifact separates ownership from display references:

• Structural strings and type: owned_context own literal contexts.
• Structural type: selector owns the contexts matched by context_prefix or collect_plugin.
• Containers such as group, flatten, grid, first_available, and view_switch aggregate context snapshots from their children.
• type: context display widgets reference contexts through contexts: but do not own them.

Generated items and placements therefore carry:

• resolved_contexts: contexts owned by that node after child and selector aggregation.
• referenced_contexts: contexts referenced by display widgets.
• unresolved_references: staged widget references that intentionally do not resolve yet, with reason, owner, expires, and item_path.

The recursive emission logic is in integrations/gen_taxonomy.py:551-719. The FE-facing meaning of the generated fields is documented in ../in-app-contract.md.

Output artifact

The generated artifact is integrations/taxonomy.json. It is validated against integrations/schemas/taxonomy_output.json and is intentionally gitignored.

Top-level shape:

{
  "taxonomy_schema_version": 1,
  "source": {},
  "sections": [],
  "placements": [],
  "opted_out_collectors": []
}

Important output concepts:

• sections[] is the resolved global section registry.
• placements[] is the ordered list of collector-owned TOC placements.
• placements[].items[] is the normalized recursive item tree.
• collector_ids links a placement back to the integration IDs produced from metadata.
• section_id is the stable registry handle; section_path is the resolved path for consumers.

Assembly, deterministic placement sorting, and output schema validation are handled in integrations/gen_taxonomy.py:847-883. Writing is handled by the generator CLI in integrations/gen_taxonomy.py:890-920.

CI flow

Pull requests run the taxonomy checker from .github/workflows/check-markdown.yml. The checker:

• validates all committed taxonomy sources by building the artifact;
• enforces taxonomy coverage when a PR changes a collector taxonomy.yaml, adds/removes it, or edits metric-bearing parts of metadata.yaml;
• runs the taxonomy unit tests.

See .github/workflows/check-markdown.yml:45-58 and integrations/check_collector_taxonomy.py.

The master regeneration workflow runs integrations/gen_taxonomy.py as part of the integrations regeneration job; see .github/workflows/generate-integrations.yml:59-68. The generated taxonomy.json is still a runtime/downstream contract artifact, not a committed source file.

Worked mental model

For a static collector such as MySQL:

1. metadata.yaml declares mysql.queries.
2. mysql/taxonomy.yaml owns mysql.queries in a structural item.
3. A summary grid widget may also reference mysql.queries.
4. The generated placement includes mysql.queries in resolved_contexts because it is owned, and in referenced_contexts where the widget uses it.

For a dynamic collector such as SNMP:

1. metadata.yaml declares a dynamic namespace such as snmp..
2. snmp/taxonomy.yaml may use a narrower selector like snmp.device_prof_ under that declared namespace.
3. Selector items own the matched context snapshot; selector references inside widgets reference dynamic contexts without claiming ownership.
4. The generated JSON preserves selector objects so downstream frontend code can resolve runtime dynamic contexts cleanly.

How I figured this out

Files read:

• integrations/gen_taxonomy.py
• integrations/check_collector_taxonomy.py
• integrations/schemas/taxonomy_collector.json
• integrations/schemas/taxonomy_output.json
• .github/workflows/check-markdown.yml
• .github/workflows/generate-integrations.yml
• ../schema-reference.md
• ../in-app-contract.md

Commands used during the original analysis:

rg -n "def module_contexts|def build_metadata_indexes|def process_taxonomy_file|def emit_item|def build_taxonomy" integrations/gen_taxonomy.py
rg -n "gen_taxonomy|check_collector_taxonomy|taxonomy.json|taxonomy.yaml" .github/workflows integrations/README.md .agents/skills/integrations-lifecycle/taxonomy.md