chartengine

chartengine compiles chart templates and builds deterministic chart plans (create, update, remove) from metrix.Reader snapshots.

Audience: ModuleV2 collector authors and framework contributors.

See also: charttpl (template DSL), metrix (metrics storage and read API).

Purpose

Collector-Facing Contract

For ModuleV2 collectors, the runtime integration expects:

Public API Surface

End-to-End Example (Single Flow)

// 1) Collector writes metrics.
store := metrix.NewCollectorStore()
meter := store.Write().SnapshotMeter("app")
meter.Counter("requests_total").ObserveTotal(100)

// 2) Engine loads chart template.
engine, err := chartengine.New(
	chartengine.WithEnginePolicy(chartengine.EnginePolicy{
		Autogen: &chartengine.AutogenPolicy{Enabled: false},
	}),
)
// handle err

err = engine.LoadYAML([]byte(`
version: v1
groups:
  - family: App
    metrics: [app.requests_total]
    charts:
      - id: requests
        title: Requests
        context: requests
        units: requests/s
        dimensions:
          - selector: app.requests_total
            name: requests
`), 1)
// handle err

// 3) Prepare plan from flattened+raw reader, emit it, then commit.
// ReadFlatten() is included even for templates with static dimensions
// because it is required for inferred dimensions, structured-family autogen,
// and is the standard pattern.
attempt, err := engine.PreparePlan(store.Read(metrix.ReadRaw(), metrix.ReadFlatten()))
// handle err
plan := attempt.Plan()
defer attempt.Abort()

err = chartemit.ApplyPlan(api, plan, chartemit.EmitEnv{
	TypeID:      "plugin.job",
	UpdateEvery: 1,
	Plugin:      "example",
	Module:      "example",
	JobName:     "example",
})
// handle err
err = attempt.Commit()
// handle err

PreparePlan Lifecycle

PreparePlan executes a deterministic phase pipeline. Terms like "materialized state" and "route cache" are defined in the Engine State section below.

Route Diagnostics and Policy Inspection

WithPlanRouteDiagnosticObserver is an opt-in validation/test surface. When enabled, the planner synchronously streams attempt-local facts for filtering, candidate rejection, chart-identity rejection, resolved routes, accepted routes, autogen displacement, collisions, lifecycle rejection, and unmatched series.

• Diagnostic planning bypasses route-cache lookup and storage so every scanned series produces complete facts.
• The default path is unchanged when no observer is configured; it continues to use the route cache.
• The callback MUST NOT call back into the same Engine and SHOULD aggregate bounded facts rather than retain every event.
• Resolved facts include the compiler template identity, series identity, rendered chart/dimension names, raw instance identity, ordered instance-label keys, and the label key used for a dynamic dimension name. Accepted facts additionally expose the effective context, family, units, algorithm, aggregation, presentation, series kind, scale, and label-promotion policy. They do not include the full input label set; a rendered chart or dynamic dimension name can itself be derived from label values and must be handled accordingly by consumers.
• ChartTemplateIDAt correlates compiler facts with a decoded template's group/chart position.

ResolveInstanceLabelPolicy is the corresponding read-only inspection helper for instances.by_labels and instances.optional_by_labels. It returns the runtime-normalized required keys, optional keys, exclusions, and include-all flag, so validation tooling does not reproduce compiler or planner precedence rules.

Reader Requirements

If inferred dimensions are present without flattened reader metadata, PreparePlan returns an explicit error.

Action Semantics

chartemit normalizes emitted action order by phase:

1. create chart/dimensions
2. update values
3. remove dimensions/charts

Routing and Collision Rules

Each metric series is routed to a chart and dimension based on template selectors. The following rules apply when routing conflicts arise:

[!WARNING] Cross-template chart ID collisions cause silent data loss — conflicting series are dropped with no error and no log entry. If metrics are missing, check for duplicate rendered chart IDs across template groups.

Authored charts can set one reducer for all their dimensions. Supported values are sum (default), min, max, and avg. Reduction is scoped to one successful plan build and happens before multiplier/divisor and absolute/incremental chart processing. Autogen routes retain their existing sum behavior. See the chart-template format's aggregation section for semantics and metric-type constraints.

Chart Label Intersection

• Promoted non-identity labels are common metadata: a key/value survives only when it is present with the same value on every routed contributor to the chart.
• An unlabeled contributor participates as an empty label set. If it shares a chart with labeled contributors, no promoted non-identity label can describe the entire chart.
• A changed intersection emits a complete replacement label set through UpdateChartLabelsAction; chart identity and dimensions are not recreated.

Authored chart-label promotion is presence-aware. Omitted label_promotion uses automatic intersection, a non-empty list uses explicit intersection over that allowlist, and an explicit empty list retains only instance identity labels. Group defaults preserve the same three states through inheritance.

Instance Identity Resolution

• instances.by_labels defines required identity. A missing explicit label rejects that series from the chart.
• instances.optional_by_labels defines declaration-ordered explicit keys that participate only when their value is present and nonblank. Missing or blank optional values retain the base/required chart identity.
• Required values render before optional key/value pairs in the chart-ID suffix. For example, present pid="1234" contributes _pid_1234. Every participating key is emitted as an identity chart label.
• Optional identity is resolved independently for each series. Present and absent source shapes can therefore materialize refined and base chart instances in the same plan; a series is routed once, never copied into an aggregate view.
• Presence/value changes are ordinary identity changes. Existing lifecycle policy expires the old chart; the engine keeps no migration or alias state.
• Explicit required/optional resolution reuses one compiled per-template plan and uses binary search over canonical sorted source labels on a route-cache miss. The existing by_labels: ["*"] path scans and sorts visible labels; wildcard identity cannot be combined with optional keys.

Algorithm Resolution

• An omitted authored algorithm remains AlgorithmAuto in the immutable program, route cache, and chart-level action metadata.
• After route lookup, the planner resolves the local route's dimension algorithm from the current SeriesMeta.Kind: counters use incremental; gauges and every other or unknown kind use absolute.
• An explicit authored absolute or incremental value overrides the runtime kind for every dimension in that chart.
• Authored and autogen routes use the same runtime resolver. Metric-name suffixes do not determine chart algorithms.
• Flattened structured families use their emitted scalar Kind. SourceKind and FlattenRole still control chart shape and identity: histogram buckets/count/sum and summary count/sum are counters; summary quantiles and StateSet states are gauges; MeasureSet fields follow their declared semantics.
• Different kinds may coexist in distinct rendered dimensions. Series aggregated into one rendered dimension must have the same kind when the chart omits algorithm; use an explicit chart algorithm for an intentional mixed-kind collapse. Chartengine does not diagnose a violation at runtime, so authoring validation and real-path tests must enforce this rule; the resulting dimension metadata is otherwise unspecified.
• A live metric identity must keep the same kind while its dimension is materialized. Kind changes are resolved for route planning, but an existing Netdata dimension keeps its creation-time wire algorithm until it expires and is recreated.

Route-cache entries are immutable discovery results. Algorithm resolution happens after cache lookup, so a cached route cannot retain the effective algorithm from an earlier series kind. Only concrete algorithms reach accumulated dimension state and dimension actions; chart-level metadata retains the configured policy.

Lifecycle Defaults and Policy

Default lifecycle policy when template omits lifecycle:

Autogen Notes

Rejected fallback series continue through collection and remain accounted as unmatched routing; no separate runtime counter is emitted. Use an upstream selector or relabeling/filtering mechanism when data, rather than only fallback charts, must be removed.

Histogram bucket charts use non-overlapping range bucket totals from metrix.ReadFlatten() and are emitted as heatmap charts in both autogen and template-driven paths. The le label remains the upper-bound identity for the bucket dimension. Autogen and template-inferred bucket dimensions are named by the bare le value (for example 0.005, 0.025, +Inf). Bucket dimensions are ordered by numeric upper bound with +Inf last, not by lexical dimension name.

MeasureSet autogen specifics:

• chartengine treats MeasureSet as a structured family, similar to StateSet, not as grouped scalar coincidence
• flattened MeasureSet inputs are expected to carry:
  • SourceKind = MetricKindMeasureSet
  • FlattenRole = FlattenRoleMeasureSetField
  • per-field metric names like <name>_<field>
  • a synthetic reserved field label (measure_field=<field>)
• the synthetic measure_field label is the authoritative field-identity channel; the per-field metric-name suffix remains for MetricMeta(name) compatibility
• gauge-like MeasureSet fields autogen with absolute algorithm behavior; counter-like MeasureSet fields autogen with incremental algorithm behavior

Reserved Flattened Label Keys

These label keys are treated specially by chartengine when consuming flattened structured-family or distribution inputs:

Notes:

• le, quantile, and measure_field are static reserved flattened-label keys in chartengine.
• StateSet is different: it does not use a global static key; it uses the base metric name itself as the synthetic flattened label key.
• These keys are part of the flatten contract between metrix and chartengine. Reusing them as ordinary user labels on those flattened inputs is not supported.

Runtime Metrics

chartengine self-instruments to a runtime store by default (disable with WithRuntimeStore(nil)).

Engine State