Prometheus profile schema and runtime effects

This file is a profile-specific navigation aid. The authoritative profile envelope is src/go/plugin/go.d/collector/prometheus/profile-format.md; the authoritative embedded chart schema is src/go/plugin/framework/charttpl/README.md. Consult the relevant sections when authoring. A Prometheus profile embeds one chart-template group, not a full standalone chart-template spec.

Contents

• Profile envelope
• Runtime capability and stock policy
• Groups are hierarchy and scope
• Charts control presentation and identity
• Instances, dimensions, and labels
• Type-dependent metric names
• Context and ID consequences
• Unsupported shortcuts

Profile envelope

match: 'exporter_*'
app: exporter

fallback_type:
  gauge: [exporter_open_connections]
  counter: [exporter_events]

relabeling:
  - match: exporter_legacy_requests_total
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: exporter_legacy_requests_total
        target_label: __name__
        replacement: exporter_requests_total

template:
  family: Exporter
  context_namespace: exporter
  metrics: [exporter_up]
  charts:
    - title: Availability
      context: availability
      units: state
      dimensions:
        - selector: exporter_up
          name: up

• match is REQUIRED and uses Netdata simple patterns against scraped Prometheus family base names. It is an exporter-detection signature, not a coverage declaration or dimension selector.
• match defines the profile's source-family selection scope. An autogen.selector rule is evaluated later against the final post-profile family name and labels, but it affects fallback only for samples originating inside that profile scope. A contributed stock profile MUST remain open to unknown future families unless a bounded source contract proves an authored alias/normalization or deliberate exclusion.
• relabeling is OPTIONAL exporter-owned normalization. It runs after profile selection and before template routing. The first applicable selected profile normalizer owns each original source family; all selected templates consume the resulting shared stream.
• fallback_type is OPTIONAL profile-owned classification for source-proven untyped scalar families. Patterns match exact post-job, pre-profile family names inside the profile's match scope. Precedence is declared TYPE, job gauge, job counter, first selected profile classification, then implicit _total counter; gauge wins over counter inside one profile. Classification is bound before profile relabeling, so a later rename preserves rather than discovers the type. Use the narrowest stable patterns.
• The runtime format supports autogen.selector.allow, but contributed profiles MUST NOT use it: an allowlist suppresses every unknown family outside the list. This syntax remains available only for deployment-owned user policy.
• autogen.selector.deny MAY suppress fallback only for exact family base names present in the source-complete inventory. Every deny MUST be exercised by a retain_writable_unrendered proof exclusion owned by the same profile and naming the same final family. Wildcard, label-only, open-ended, fixture-absent, stale, and cross-profile denies are forbidden. A selector containing {...} is label-constrained policy, not an exact metric name. The collector store retains the samples; only generic charts disappear.
• Current source-complete evidence MUST still produce zero generic fallback and zero unmatched series. The validator tests future compatibility separately with a synthetic family derived from wildcard match scope, so an unknown future family remains generic without masking current profile incompleteness.
• Recommended-job selector denies MUST name exact exposition metric names present in the source-complete sample inventory. Dynamic source-proven alias grammars belong in bounded profile or job relabel drop rules; inverse keep/keepequal filters are forbidden for contributed profile and job pipelines. A non-empty job allow list MUST copy every positive wildcard term from profile.match into an unconstrained expression, copy the complete match expression, or use *; finite canaries and label constraints are not structural namespace coverage. Label-dependent sample discard MUST use an exact source-fixture metric block, and every exact-scope discard rule MUST actually drop a fixture sample at its ordered pipeline position. Wildcard blocks may discard only from __name__, with either a finite exact-name regex or one non-empty internal entity key between finite exporter prefixes and finite terminal metric suffixes. Every finite exact name or prefix/suffix branch MUST drop source-complete fixture evidence; open-ended terminal regexes and wildcard dropequal are forbidden. Before discard, that name MUST remain derived exclusively from the original metric name across all earlier relabel rules and blocks. A wildcard block also MUST NOT rewrite __name__ from application labels because the mutation can itself rename or invalidate unknown future families. The exact-scope exception cannot infer name-only provenance from labelmap.source_labels: Netdata's relabel runtime ignores that field and maps application-label names. The exception applies only before any earlier metric-name write; a rewrite can route a future family into an apparently exact later block. Dynamic targets are evaluated with their regex and replacement together: a finite output language that cannot produce __name__ is label-only, not a metric-name rewrite. Every positive wildcard term in a block that can discard or rename a metric MUST yield an in-scope synthetic probe that the ordered pipeline actually processes; one harmless term cannot cover another term whose exclusions hide every probe. After the pipeline, each primary future probe MUST still reach generic fallback or a bounded source-proven authored destination, MUST NOT accidentally collide with an authored dimension metric name, and MUST retain the identity promised by the declared transformation. Name provenance follows reachable ordered paths, including negative terms; a provably disjoint exact mutation does not taint a later exact block. Scope proof includes the complete glob language, including character classes. A subsequent name-only rewrite inherits any application-label provenance in its incoming __name__; it cannot clear taint.
• A wildcard name-derived replace MUST use the same finite exact-name or internal-dynamic-key-plus-terminal-suffix grammar and fixture evidence as a wildcard name drop. An internal-key rewrite MUST NOT reference the dynamic capture in its output; its finite output branches MUST all be authored canonical metric names. Before either dynamic rewrite form, an earlier replace in the same block MUST copy unchanged from __name__ the capture that encloses the entire dynamic entity region into a reserved static non-__name__ label; a nested capture covering only one alternative is incomplete. The target MUST be absent from source-fixture block inputs and preserved through every reachable later rule/block. A later label write sourced only from __name__ is harmless when its regex is disjoint from all possible current names. Canonical outputs MUST keep every finite prefix/suffix branch distinct. One rewrite-only form is also allowed for a source-owned dynamic identity tail: <canonical_name>_<non-empty-identity> may rewrite exactly to <canonical_name>, and the source-complete fixture MUST reach that branch. An unrelated terminal catch remains open-ended and forbidden.
• Across exact and wildcard blocks, the complete recommended relabel pipeline MUST preserve every observed writer-admissible logical identity after normal histogram/summary assembly. Distinct source identities MUST NOT converge on the same final metric name and labels; label removal and name normalization do not aggregate the values they collapse. Writer-rejected samples such as non-finite scalars do not participate in this identity proof.
• Every source-fixture sample reached by a metric-name rewrite MUST retain a valid non-empty name. Empty/invalid replacement output is an implicit discard, not a rewrite; intentional exclusions belong in explicit bounded, source-evidenced drop rules.
• Ordered name-rewrite reachability MUST preserve literal replacement prefixes and suffixes around captures; collapsing every capture-bearing replacement to a prefix-only wildcard creates false reachability into disjoint later blocks.
• Prefer exporter-unique family patterns in match. Generic runtime families such as process_*, python_*, and http_* may be charted without being part of detection; including them can make unrelated endpoints eligible.
• Exact-mode validation forces the candidate by name, so its PASS does not prove that automatic profile selection is unique or safe.
• app is OPTIONAL and must match the profile-name syntax. The resolved job app becomes the application segment in contexts. Precedence is configured job app, then the first selected profile app, then job name.
• Stock application metadata examples omit job profiles and use default automatic selection. They also omit job app when the selected profiles provide one unambiguous identity. Proof descriptors keep explicit supporting profiles because isolated validation composition is evidence, not job policy.
• template is REQUIRED and must contain at least one chart. Its value is a recursive charttpl.Group.
• The file basename is profile identity. Use lowercase letters, digits, and underscores, starting with a letter.
• Unknown keys fail strict decoding.

The collector supplies the full chart-template version, engine, and prometheus[.<app>] context namespace. Do not put a standalone-spec version, top-level groups, or engine beside match in a profile.

Runtime capability and stock policy

The chart-template runtime is a general user-facing format. Stock contributions use a stricter minimal policy so intent is reviewable and defaults cannot drift:

These are contribution rules, not parser limitations. Deployment-owned user profiles may use the full runtime surface when its consequences are intended.

Groups are hierarchy and scope

A group can contain:

family: Requests
context_namespace: requests
metrics: [exporter_requests_total]
chart_defaults:
  label_promotion: [region]
  instances:
    by_labels: [service]
charts: [...]
groups: [...]

Each field affects a different part of the dashboard:

• family composes with ancestor families using /. This is navigation, so use application functions and entity levels rather than metric types.
• context_namespace composes with . into chart contexts. It is identity, not merely display text.
• metrics authorizes exact metric names for dimension selectors in the group and descendants. It does not route, chart, keep, or drop a series.
• chart_defaults reduces repetition. The nearest group default wins; a chart-local value replaces it. Lists and objects are replaced, not merged.
• groups can nest to any depth. Put a metric at the highest scope that really needs it, not automatically at the root.

Why scope matters: a broad root declaration lets unrelated subtrees select the same series accidentally. A narrow declaration makes the intended ownership visible and lets the compiler catch cross-subsystem mistakes.

Remove declarations that no authored dimension in their scope uses. They do not preserve denied or writer-rejected data; they only imply ownership and can make a stale or excluded family look intentionally covered.

Charts control presentation and identity

Required chart fields are title, context, units, and at least one dimension. Stock Prometheus profiles MUST omit priority.

Optional fields include:

• id: base chart ID; otherwise derived from context by replacing dots with underscores.
• family: a chart-level family leaf.
• algorithm: absolute or incremental, applied to every dimension in that chart.
• aggregation: sum, min, max, or avg, applied when several source series render to the same chart and dimension.
• type: line, area, stacked, or heatmap.
• label_promotion, instances, and lifecycle.

An omitted algorithm is resolved independently for each rendered dimension from the matched runtime series kind: counters use incremental; gauges and other kinds use absolute. Metric-name suffixes do not select the algorithm. Distinct dimensions in one chart may therefore use different algorithms.

Set algorithm explicitly for an intentional runtime-kind override. The generic runtime also permits it to resolve differently typed series collapsed into one rendered dimension, but stock profiles must preserve distinct gauge/counter dimensions instead. Do not force metrics with incompatible units or semantics into one chart merely because a chart-wide algorithm compiles.

Priority behavior:

• omitted or non-positive priority becomes 70000;
• every Prometheus chart intentionally uses that same runtime value;
• YAML position does not become runtime or UI presentation order;
• UI sorting is outside the profile contract.

Therefore omit priority. Keep source order coherent for review without claiming that it controls presentation.

Presentation rules checked by the profile-validation tool:

• Every chart needs at least one visible dimension. Hidden dimensions MAY support a visible comparison but MUST NOT make the entire chart invisible.
• A chart selecting a histogram _bucket MUST use units: observations/s. It MUST use algorithm: incremental or omit the algorithm so the flattened counter kind selects incremental. Omit type: heatmap in stock YAML because the compiler derives it.
• Explicit area requires deliberate filled-magnitude intent.
• Explicit stacked requires an exact disjoint, exhaustive, additive partition of one whole. Units alone cannot prove or reject either relationship, so the validator emits a semantic-review warning rather than guessing from words.

Instances, dimensions, and labels

instances.by_labels

• No instances means one aggregate chart instance.
• Explicit labels create one chart per unique combination and are required on every routed series.
• * uses every label in identity; future exporter labels can therefore create new chart identities and cardinality.
• !label excludes a key; excludes win regardless of order.
• An instances block needs at least one positive token.

instances.optional_by_labels

• Optional identity keys are used only when their value is present and nonblank. Missing or blank values keep the base chart identity.
• Each present optional key contributes both its key and value to the chart-ID suffix, so partial presence remains distinguishable.
• Optional keys are explicit only: * and !label are invalid. They cannot overlap required or excluded keys and cannot accompany by_labels: ['*'].
• Optional identity does not duplicate the stream into base and detailed charts. One series routes to exactly one identity: base when absent, refined when present.
• Use it only for a bounded, sufficiently stable, operator-useful axis. It still multiplies chart cardinality and identity churn when values change.

Use instance labels for the entity the operator selects: server, database, table, backend, queue, endpoint, and so on. If a required identity label is absent, the series does not route to that chart; the validator reports the effective inherited identity, selector, and missing label keys directly. Different raw identity values may also normalize to the same chart ID, so validate observed values.

A finite or bounded category MAY be instance identity when filtering and grouping that category is more useful than comparing it as dimensions in one chart. Python GC generation uses instances.by_labels: [generation] and one fixed dimension per chart. Netdata can aggregate those charts for a runtime-wide view, so do not duplicate the aggregate chart.

The semantic instance type and its complete unique key are not always the same list of concepts. An ownership label such as database does not turn a table context into a database-table context, but {database, table} may both be required in by_labels when table names repeat. Ownership and descriptive labels that are not required for uniqueness belong in promoted chart metadata.

For required explicit identity, every routed series must provide each required label. Use selector guards when the metric can also appear without them. For optional identity, do not add such guards merely to force a detailed view: the base/refined behavior is the feature's contract.

Treat nested identities as a lattice: a child entity retains its parent's identity labels and adds the labels required for the narrower entity. Because group and chart defaults replace lists rather than merging them, a child by_labels override must repeat the parent labels deliberately. Charts that compose to the same final family path should use one effective identity set; otherwise the displayed leaf cannot filter as one entity type.

Dimensions

Every dimension needs a selector with an explicit metric name. The metric must be visible through metrics scope. Label matchers narrow the selected series.

Choose exactly one naming mode:

• name: one static dimension name;
• name_from_label: a bounded label value becomes the dimension name;
• omit both only when chartengine can infer a histogram bucket (le), summary quantile (quantile), or supported state-like dimension.

A missing or empty name_from_label value makes that route unroutable. Dynamic dimension labels should describe comparable aspects, not unbounded entities; otherwise one chart becomes an unreadable cardinality sink.

When authoritative exporter source or configuration proves that it can omit a valid dimension label, the chart MUST cover both label states with mutually exclusive selectors. Do not infer optionality merely because a chart uses name_from_label:

dimensions:
  - selector: 'requests_total{status=~".*[^[:space:]].*"}'
    name_from_label: status
  - selector: 'requests_total{status!~".*[^[:space:]].*"}'
    name: unclassified

In the Netdata selector implementation, the positive matcher selects a present value containing at least one non-whitespace character, and the negated matcher also selects a missing label. The fixed fallback therefore covers missing, empty, and whitespace-only values in the same context and chart instance without overlapping the dynamic dimensions. Do not use .+ here because chartengine trims and rejects whitespace-only dynamic names after selector matching.

• The fallback MUST describe the missing classification; it MUST NOT be named total, because it is only the unlabeled subset when labeled and unlabeled series coexist.
• The fallback name SHOULD be outside the authoritative dynamic value domain.
• A broad static selector MUST NOT accompany the dynamic selector in the same chart; the labeled series would enter both dimensions.
• This pattern does not make a missing entity identity available. Use instances.optional_by_labels when one source legitimately has a stable base identity without that key and a refined identity when it is present.
• This pattern does not authorize invalid gauge aggregation. Preserve the complete identity for non-additive states.

options supports integer multiplier, divisor, hidden, and float. Conversions affect values but do not change source semantics. Use a negative multiplier only for a deliberate mirrored visualization, such as inbound versus outbound traffic.

Label promotion

Promoted labels are mutable chart metadata for filtering/grouping; they are not identity and do not split chart instances. The field has three states:

• omitted: automatically promote non-identity labels whose value is identical across every series contributing to the chart;
• non-empty: promote only listed non-identity labels whose value intersects;
• []: promote no non-identity labels.

An unlabeled contributor makes the promoted intersection empty. Required and present optional identity labels are always chart identity metadata. Promote only stable descriptive labels. Do not expect promotion to rescue an _info family that the Prometheus writer never materializes.

Aggregation

Aggregation is used only when multiple routed series render to the same chart ID and dimension. It does not reduce scrape/store cardinality.

• sum: additive counters, histogram components, or source-defined additive gauges;
• avg: an unweighted typical value for a non-additive gauge when that mean is meaningful;
• min/max: extrema or a deliberate reduction of status-like 0/1 values.

The default is sum. A chart uses one reducer for all dimensions. Summary quantiles cannot be merged into a global quantile; preserve identity instead. avg is unweighted, and a non-sum reducer over cumulative counters can create misleading deltas when contributor membership changes. In stock YAML, state an aggregation explicitly when the design deliberately allows many-to-one collisions, including deliberate sum; omit it for collision-free routes.

Lifecycle

max_instances, expire_after_cycles, dimensions.max_dims, and dimensions.expire_after_cycles manage dynamic cardinality. Caps are best-effort and can omit observed entities when undersized. Use them as a cardinality policy with known consequences, not as a substitute for choosing the right identity/dimension model. The validator rejects a cap that discards entities or dimensions present in the supplied dump: a mechanical PASS must not hide current evidence. A cap above observed cardinality still requires reasoning about configurations and future values absent from the dump.

Type-dependent metric names

The profile selects the flattened names exposed to chartengine:

• gauge/counter: family name;
• histogram: <base>_bucket, <base>_count, <base>_sum;
• summary: <base> for quantiles, plus <base>_count and <base>_sum.

Declare every flattened name a dimension selector uses. The bare histogram base is not a flattened scalar. A summary base exists only when the source provides quantiles; count and sum remain available for a valid quantile-free summary. See metric-types.md for writer and design behavior.

Context and ID consequences

The effective context is built from:

prometheus.<resolved-app>.<group namespaces...>.<chart context>

When the profile root namespace equals the resolved app, the collector removes that duplicate segment. Context and chart ID are durable identity surfaces: renaming them creates new chart identities and can leave old metadata/history to expire separately.

Two templates can derive the same rendered ID even when YAML paths differ. The whole planner keeps one owner, so validate isolated charts and observed instance values with the repository validator. Chart IDs, contexts, and dynamic dimension names are sanitized again at public wire emission. Empty values and cases where distinct effective values collapse to one wire value are objective failures; intentional reuse of the same raw context remains a dashboard-design judgment rather than a blanket uniqueness error.

Unsupported shortcuts

The profile schema has no separate top-level drop: shorthand and no computed average dimension. Use profile relabeling for exporter-owned exclusions after selection; use the job selector or job relabeling for deployment policy or exclusion needed before selection. Chart only values the collector actually writes; do not document or emit speculative schema keys.