Prometheus profile validation

This developer tool validates one candidate go.d/prometheus.profiles file, with optional supporting profiles, against one captured Prometheus exposition dump. It exercises the production profile catalog, Prometheus collector, writer, metrix store, chart-template compiler, chartengine planner, and public chart emitter instead of reimplementing those contracts.

The tool runs with go run; it is not installed and is not a go.d.plugin mode.

See the canonical framework architecture for the relationship between standalone validation, stock semantic proofs, production diagnostics, and external testdata. This document owns the standalone CLI, job policy, findings, and extension details.

Run

From src/go:

go run ./tools/prometheus-profile-validation \
  --profile /path/to/exporter.yaml \
  --support-profile /path/to/runtime.yaml \
  --dump /path/to/metrics.txt \
  --job /path/to/job-policy.yaml \
  --output text

--support-profile is optional and repeatable; it supplies profiles expected to compose with the candidate on the same endpoint. --output accepts text (default) or json. --job is optional for exploration, but a deliverable profile needs the intended job policy; omitting it produces a warning because collector defaults may not match deployment.

Stock-profile fixtures and strict source-semantic contracts live in netdata/testdata. Clone its latest master into the ignored src/go/testdata directory, or set NETDATA_TESTDATA_DIR to another checkout. Each profile uses one stable external directory; update latest testdata and the corresponding Netdata proof expectations together. For an existing default checkout, shallow-fetch origin master and detach at the fetched tip before replay. Local feature branches remain unchanged.

git -C src/go/testdata fetch --depth=1 origin master
git -C src/go/testdata switch --detach FETCH_HEAD

The dedicated replay checks every declared outcome and reconciles source semantics, normalization, routes, chart plans, observations, wire identities, and aggregate semantic coverage. Required mode fails when external evidence is unavailable:

NETDATA_PROMETHEUS_TESTDATA_REQUIRED=1 go test -count=1 \
  ./internal/promprofile/validation -run TestStockProfileProofsReplay

go run ./tools/prometheus-profile-proof verify --repo-root ../.. verifies the complete stock-proof catalog, exact local and external layouts, strict source/profile contracts, and executable replay expectations.

Exit codes:

• 0: objective validation passed; warnings may still require review.
• 1: validation completed and found an objective failure.
• 2: command-line usage or report-output failure.

Architecture and extension points

The executable is a flag/exit-code adapter over promvalidation.Validate in internal/promprofile/validation. Contributor policy, deterministic findings, and report rendering live in that internal library. Runtime semantics remain with their production owners:

• pkg/prometheus parses/assembles evidence and defines physical/logical series identities plus strict duplicate detection;
• the Prometheus collector performs selector, relabel, writer, exact-profile, and store processing and emits opt-in structured pipeline facts;
• pkg/matcher and collector relabel own bounded, cancelable witness/name-flow mechanisms used by contributor policy;
• chartengine emits opt-in route facts from its one production plan; and
• chartemit exposes structured public-ID/context/dimension inspection from the same preparation path used by application.

Diagnostics are constructor options, not collector configuration. Disabled production paths do not allocate diagnostic records. Enabled validation bypasses the chart route cache so every series produces an attempt-local fact; observers stream or aggregate within one run and retain no process-global state.

Add a new semantic fact at the production owner and expose a small structured diagnostic/mechanism. Add contributor severity, evidence obligations, wording, and report presentation in internal/promprofile/validation. Do not reconstruct runtime acceptance, identity, naming, or wire behavior in the validator.

After isolated inputs and profiles are loaded, scraping, collector execution, future-input replay, matcher/relabel analysis, and provenance finalization use a 30-second deadline bounded by the caller context. Input staging, profile decode, authored-template traversal, and the chartengine plan call are synchronous and are not covered by that internal timeout. Matcher/relabel analyzers also have deterministic state/transition/value limits and fail closed when those limits or context cancellation are reached.

Safe job policy

The job file accepts only settings that shape the captured dump:

name: exporter
selector:
  allow: ['exporter_*']
  deny: [exporter_requests_created]
relabeling:
  - match: exporter_*
    metric_relabel_configs:
      - source_labels: [old_label]
        target_label: instance
        action: replace
fallback_type:
  gauge: ['exporter_*']
  counter: ['*_total']
expected_prefix: exporter_
max_time_series: 2000
max_time_series_per_metric: 200
future_inputs:
  - name: exporter_future_requests_total
    type: counter
    labels:
      instance: future-instance

The schema has no URL, authentication, TLS, or profile-selection fields. The tool snapshots the dump, forces an isolated file:// endpoint, empties ambient profile catalogs, and selects exactly the supplied candidate/supporting set. Unknown job keys and multiple job documents fail.

app remains an accepted deployment-shaped field for an intentional override or for disambiguating profiles that declare different apps. Omit it when automatic profile selection already supplies the intended identity. Stock proof composition is declared by PROFILE-DESIGN.yaml and resolved through production automatic selection; it is never copied into job policy.

future_inputs are validator-only raw samples. name is required; type is optional (gauge by default, counter, or untyped); and labels is an optional string map that cannot contain __name__. Names and label names use the production Prometheus UTF-8 rules. A job may declare at most 256 unique raw identities and one type per family. Use invented, non-sensitive labels. These samples enter before job selector processing and both job and profile relabeling, so do not declare post-relabel names.

The validator derives bounded raw witnesses when metric-name identity is preserved. If a reachable job or profile relabel rule can write metric names, explicit future_inputs are required because arbitrary relabeling cannot be inverted soundly.

What PASS establishes

For the supplied dump and job policy, a pass establishes that:

• strict catalog/profile/template decoding succeeds;
• the real collector completes Init, Check, and a committed Collect;
• one production chart plan emits complete structured route diagnostics for every scanned writer series;
• every current writer series is accounted for with zero generic fallback and zero unmatched series;
• a second isolated real collector sequence introduces declared or derived raw future probes before the job selector and job/profile relabeling; future results cannot satisfy current coverage;
• every positive wildcard term and future-relevant relabel scope/rule in the candidate and each supplied support profile has a probe that actually traverses the production selector, relabeler, assembler, writer, exact profile selection, and planner;
• future probes neither overwrite current writer identities nor collapse with each other, and reach generic fallback unless a bounded contributor-approved alias normalization legitimately maps them to an authored family;
• any non-empty job allow list structurally covers each positive wildcard term in the profile scope, and wildcard relabel discard has a bounded, source-fixture-evidenced metric-name grammar without prior label-derived metric-name mutation; wildcard metric-name rewrites are derived only from the original name and use a bounded, source-fixture-evidenced input grammar;
• every supplied profile has no fallback allowlist or open-ended fallback deny, and neither job nor profile relabeling has a sample-filtering keep or keepequal action; exact profile/job/relabel exclusions have the required source-complete evidence; a selector containing {...} is label-constrained policy rather than an exact metric name;
• the complete job-then-profile relabel pipeline preserves every observed writer-admissible logical identity after normal histogram/summary component assembly; distinct source identities do not converge on the same final metric name and labels;
• every source-fixture sample reached by a metric-name rewrite retains a valid non-empty name rather than being implicitly discarded;
• every authored chart and every authored dimension materializes;
• every selected writer series carries every label required by its chart's effective explicit instance identity;
• observed cross-template rendered-ID collisions are absent;
• observed same-template instance identities do not collapse into fewer chart IDs;
• observed per-instance dimension identities are not discarded by lifecycle caps or planner normalization;
• planner-distinct chart IDs remain non-empty and unique after public wire normalization;
• distinct effective contexts do not collapse into one public wire context, and effective contexts do not normalize to empty;
• observed dynamic dimension names do not collapse into duplicate emitted wire IDs or sanitize to an omitted empty ID;
• every chart omits explicit priority so all use the common runtime default;
• every chart has at least one visible dimension;
• histogram bucket charts use observation-rate units and incremental semantics.

Explicit area and stacked are advisory rather than objective failures. Units cannot prove the relationship: area needs deliberate filled-magnitude intent, while stacked needs an exact disjoint, exhaustive, additive partition of one whole. The strict stock proof records and verifies that semantic intent.

instance_identity_label_unavailable reports the direct cause when a chart's effective inherited/explicit instances.by_labels requires keys absent from writer series selected by one of its dimensions. It reports only the chart, selector, missing label keys, and affected count—not observed label values. Changing the entity boundary or overriding identity can be correct; inventing a label that the series does not carry is not.

pipeline_excluded records raw source families that are wholly or partly absent after job/profile policy and writer processing. Each record carries the raw and materialized logical-series counts plus every matched mutation/drop rule in policy_paths; counts.pipeline_excluded is the number of family records, not the sum of lost logical identities. Relabel-renamed families are reconciled by logical identity against their final writer names, so successful normalization is not reported as loss. Categories are deliberately generic when selector or relabeling rules make the precise cause ambiguous. Raw logical series and flattened writer series are different counts for histograms and summaries.

pipeline_renamed reports successful raw-to-final family-name lineage separately, including the raw logical-series count and how many renamed identities reached the writer. Its policy_paths identify the exact job/profile rules that changed the source identities. A family can appear in both sections when only part of its source identity set survives normalization.

profiles.candidate, profiles.supports, and profiles.selected distinguish the authored input roles from the profiles production selection actually activated. When validation composes profiles, profile-owned chart and relabel paths use profiles[<name>].template... and profiles[<name>].relabeling...; candidate-only validation retains its existing concise paths.

The strict current-source requirement is zero autogen and zero unmatched series. One explicit profile policy can pass with a warning:

• profile_suppressed_series means production route facts show every unmatched series was suppressed by selected profiles' explicit fallback selectors. Each contributing profile must have an exact deny list. Owner-qualified finding paths identify the candidate and/or supporting selectors involved. This proves the reported gap is an intentional deny policy; it does not prove that suppressing the operator question is a good design.

Forward compatibility combines bounded structural analysis with the separate raw future-input run. Wildcard name-discard and name-rewrite exceptions also require the current source-complete dump to exercise every bounded grammar branch. The profiles.candidate.future_raw_probe and profiles.supports[].future_raw_probe fields record each profile's first open raw probe that actually covers one of its wildcard scopes or relabel rules. Declared input order does not assign ownership. Shared job policy and all profile-local requirements are exercised by one composed future collector/planner run. Current physical sample names and logical typed-family base names are excluded from accepted and derived probes. These objective errors reject closed contribution policy even when the current fixture has no coverage gap:

• closed_profile_fallback: non-empty autogen.selector.allow;
• open_ended_profile_fallback_deny: wildcard, label-only, or otherwise non-exact profile deny;
• unproven_profile_fallback_deny: an exact profile deny names a family absent from the source-complete family inventory;
• open_ended_job_selector_deny: wildcard, label-only, or otherwise non-exact recommended-job deny;
• unproven_job_selector_deny: an exact recommended-job deny names an exposition metric absent from the source-complete sample inventory;
• closed_job_selector_allow: a non-empty job allow list does not contain each positive wildcard profile term unchanged in an unconstrained expression, the complete profile match expression, or *;
• future_metric_witness_unavailable: wildcard profile scope cannot produce a valid deterministic Prometheus-family probe;
• future_relabel_witness_unavailable: one positive wildcard term in an overlapping relabel block can discard or rename metrics, but its exclusions hide every deterministic in-scope probe or ordered relabeling prevents the block from processing those probes;
• future_inputs_required: reachable relabeling can change metric names but no explicit raw probes were declared;
• future_input_not_future: a declared probe name already occurs in current evidence;
• future_profile_term_uncovered, future_relabel_scope_uncovered, and future_relabel_branch_uncovered: the actual future run did not cover one required positive wildcard term, relabel scope, or rename/drop rule;
• future_metric_blocked_by_profile: a future writer series is unmatched by chart routing;
• future_metric_blocked_by_job_selector: the recommended job rejects it;
• future_metric_blocked_by_job_relabel: an otherwise applicable relabel rule drops it;
• future_metric_blocked_by_profile_relabel: a selected candidate or supporting profile's relabeling drops it;
• future_metric_rejected_by_writer: the production writer rejects its final family or series identity;
• future_metric_routed_to_authored_metric: relabeling maps a primary future probe onto a metric selected by an authored dimension without a recognized bounded alias contract;
• future_metric_identity_collapse: relabeling maps distinct primary future probes onto the same writer identity, or a future probe overwrites a current identity, which can merge unrelated families or types;
• future_run_changed_current_evidence: adding raw future probes changes or removes a current writer identity/value;
• observed_relabel_identity_collapse: after normal histogram/summary component assembly, the complete job-then-profile relabel pipeline maps multiple observed writer-admissible source identities onto the same final metric name and labels. Relabeling retains one value for that final identity; it does not aggregate the collapsed sources. Writer-rejected samples such as non-finite scalars do not participate;
• invalid_relabel_metric_name_discard: a source-fixture sample is implicitly dropped because recommended relabeling produces an empty or invalid metric name. Intentional exclusions must use explicit bounded, source-evidenced drop rules;
• closed_relabel_filter: a relabel rule uses inverse keep or keepequal;
• unbounded_relabel_discard: a wildcard relabel block uses application labels to discard samples instead of binding the exclusion to __name__; label-based discard requires an exact known metric block;
• open_ended_relabel_name_discard: a wildcard __name__ discard is not a finite exact-name set or one non-empty internal entity key between finite exporter prefixes and finite terminal metric suffixes. Wildcard dropequal is also rejected because it cannot regex-bound the discarded names;
• unproven_relabel_name_discard: at least one finite exact name or dynamic alias prefix/suffix branch has no dropped sample in the source-complete fixture;
• unproven_exact_relabel_scope: an exact positive relabel-block metric name is absent from the source-complete exposition;
• unproven_exact_relabel_discard: an exact-scope drop or dropequal rule drops no fixture sample at its ordered pipeline position;
• tainted_relabel_name_discard: an earlier ordered relabel rule may derive __name__ from application labels before a later sample-discard rule reads it. Name provenance is tracked across relabel blocks;
• unbounded_metric_name_rewrite: a wildcard relabel block may write __name__ from application labels, including through a labelmap replacement that can create __name__. The rewrite itself can rename or invalidate an unknown future family. Netdata's relabel runtime ignores configured labelmap.source_labels, so naming only __name__ there does not establish name-only provenance. An exact block is bounded only while the name is still the original metric name; earlier renaming can route future families into that block. Provenance follows reachable output scopes using exact glob-language intersection, so a disjoint exact or character-class rename does not poison a later block. Capture-bearing replacements retain both literal prefixes and suffixes in that reachability proof. A name-only rewrite inherits any application-label provenance in its input __name__. Dynamic destination capability is evaluated from the rule regex and replacement together, so a finite output set that cannot produce __name__ remains valid label-only policy;
• open_ended_relabel_name_rewrite: a wildcard name-derived replace does not enumerate finite exact inputs, one internal dynamic entity key with finite terminal suffixes, or the rewrite-only canonical dynamic-tail form <canonical_name>_<non-empty-identity> -> <canonical_name>. An internal-key rewrite also fails when its output references the dynamic capture, a finite capture nested inside that identity, or any finite output is not an authored canonical metric;
• unpreserved_relabel_name_identity: an internal-key or canonical-tail rewrite removes the dynamic identity from __name__ without an earlier rule in the same block copying unchanged the capture that encloses the entire dynamic entity region into a reserved stable non-name label. A nested capture covering only one alternative is incomplete. The target must be absent from fixture inputs at block entry and survive every reachable later rule/block. The canonical output must keep every finite prefix/suffix branch distinct. A later label write sourced only from __name__ is ignored when its regex is disjoint from every possible current canonical name. Distinct raw entities could otherwise collapse into one writer series;
• unproven_relabel_name_rewrite: at least one finite name, prefix/suffix branch, or canonical dynamic-tail prefix does not reach the rewrite in the source-complete fixture.

authored_mapping reports charts from the selected candidate/support composition in source order:

• the source profile-local chart path for composed validation;
• composed displayed family;
• title, context, units, authored algorithm intent, and type;
• effective instances.by_labels after inheritance; and
• exact dimension selectors, static/dynamic naming mechanisms, and visibility.

This mapping is deterministic evidence for semantic review. It lets the author reconcile the emitted selector-to-display design against a source-backed operator model without claiming that the validator understands the application's causal structure. Materialized chart records separately report the compiler-resolved runtime algorithm.

Rendered chart IDs and dimension names can contain label values, so reports fingerprint them instead of emitting the raw values. Public-emitter failures are classified and fingerprinted rather than copied verbatim for the same reason.

Warnings preserve design judgment

Warnings identify designs that deserve explanation but can be correct:

• exact explicitly suppressed fallback series, whose source boundary and operator trade-off require review;
• profile match expressions that also accept common go_*, http_*, process_*, or python_* families and can therefore auto-select on an unrelated endpoint;
• open-ended instances.by_labels: ['*'] identity;
• repeated non-empty sibling family names, which compose the same displayed navigation path instead of distinct semantic branches;
• displayed leaf families whose charts use different effective entity identities;
• child group/chart identities that drop labels from an explicitly declared parent chart_defaults.instances.by_labels identity;
• sibling family subtrees with no common explicit identity label;
• observed label keys that selected chart series carry but the chart neither uses nor explicitly excludes for identity, dimensions, promoted metadata, or selector routing;
• one union summary of the observed writer-capable evidence removed by the effective job selector before profile coverage is measured;
• each allow list's observed exclusion of otherwise writer-capable families;
• each deny expression's observed impact on otherwise writer-capable families;
• every sample-discarding job or profile relabel rule, including its observed logical-identity impact or the absence of matching evidence; keep and keepequal additionally fail contributor validation because they are closed inverse filters;
• metric declarations unused by any authored dimension in their scope;
• histogram bucket charts whose authored type differs from the compiler-forced heatmap;
• every explicit area chart, so deliberate filled-magnitude intent is reviewed;
• every explicit stacked chart, so the exact disjoint/exhaustive/additive relationship is reviewed;
• charts that mix distribution shape/count/sum roles under one unit;
• compiler-resolved incremental charts whose rendered units omit the per-second denominator and are not a recognized derived equivalent such as cores, concurrency, or utilization;
• absolute chart dimensions whose observed non-zero magnitudes differ by at least 20x and may flatten the smaller signal.

The validator does not turn these heuristics into policy. For example, sibling families can intentionally represent different entity levels, and a leaf can mark an intentional filter boundary. Identity findings report effective label keys only; they cannot determine the application's entity types or invent a missing label. A label can be intentionally aggregated, but its lost comparison/filtering and expected cardinality still need explanation. Presentation likewise depends on the operator question and source relationships, not a unit-word allow/deny list.

The exclusion summary prevents a mechanically clean report from hiding a shrinking denominator among many per-rule warnings. It remains advisory because the tool cannot prove why the user chose a collection boundary. Dashboard focus alone is not a collection boundary: use hierarchy and coherent chart composition rather than discarding distinct writer-capable diagnostics.

Relabel discard warnings consume facts emitted while the production processor runs the validated ordered blocks and attribute observed drops to the exact rule. A rule with no observed drop still warns because one dump cannot prove its future exclusion surface. Under an overlapping wildcard stock-policy scope, an otherwise permitted __name__ drop or rewrite also fails unless every bounded grammar branch has evidence. The warning never reports observed label values.

Incremental-unit warnings use chartengine's runtime-kind-resolved algorithm rather than inferring an algorithm from metric-name suffixes. Compound observed units such as duration/item/s retain rate semantics and are not rejected merely for having more than one denominator.

Hidden dimensions are excluded from the visible-scale comparison because they do not control the chart's default visible axis. A chart may hide supporting dimensions, but hiding every dimension is an objective failure because the chart has no visible answer. The discoverability of each remaining hidden dimension is still a semantic review question.

Evidence boundary

This is an objective correctness gate, not a dashboard designer:

• One dump cannot validate metrics, optional features, labels, or label values absent from it. Use a comprehensive representative dump. Raw future probes prove only that declared/derived identities traverse current policy; they do not invent or validate a future metric's semantics, cardinality, or real labels.
• Observed ID collisions are checked; future unseen values can still normalize to the same ID.
• A lifecycle cap that accommodates this dump may still omit entities or dimensions in a larger configuration.
• The tool forces exact selection of the supplied candidate/supporting set. It does not prove that their match expressions uniquely auto-select this exporter instead of an unrelated endpoint. The generic-match warning catches common unsafe classes, not every possible false-positive signature.
• Writer-rejected source series are reported but are not profile coverage gaps, because chartengine never receives them.
• A PASS enforces only the units and presentation cases listed above. It does not establish that the remaining hierarchy, chart composition, titles, units, instance choices, or presentation order are useful to an operator.

Each validation call constructs isolated current and future collectors with an explicit candidate catalog. It does not mutate production catalog or plugin configuration globals, so library tests run in-process.