Back to Netdata

chartengine

src/go/plugin/framework/chartengine/README.md

2.10.315.6 KB
Original Source

chartengine

chartengine compiles chart templates and builds deterministic chart plans (create, update, remove) from metrix.Reader snapshots.

Audience: ModuleV2 collector authors and framework contributors.

See also: charttpl (template DSL), metrix (metrics storage and read API).

Purpose

StageResponsibility
charttplTemplate decode/defaults/validation
chartengine.CompileBuild immutable program IR
Engine.PreparePlanPrepare plan actions plus explicit commit/abort boundary
chartemit.ApplyPlanEmit plan to Netdata wire protocol

Collector-Facing Contract

For ModuleV2 collectors, the runtime integration expects:

RequirementWhy it matters
MetricStore() returns metrix.CollectorStore (cycle-managed)Job runtime controls cycle boundaries and success/failure semantics
ChartTemplateYAML() returns valid charttpl YAMLLoaded once at autodetection/post-check
Collector writes metrics during Collect() onlyPlanner runs after a successful cycle commit
Metric names used in template selectors are present in group metric scopeCompile/validate consistency

Public API Surface

APIPurpose
New(opts...)Create engine with policy/runtime options
Load(spec, revision) / LoadYAML(data, revision)Compile and publish program revision
PreparePlan(reader)Build deterministic action plan from reader snapshot and return an explicit attempt
RuntimeStore()Access chartengine internal runtime metrics store
WithEnginePolicy(...)Configure selector + autogen behavior
WithRuntimeStore(...)Override/disable self-metrics store
WithSeriesSelectionAllVisible()Process all visible series instead of filtering to latest successful collect cycle. Intended for runtime/internal stores that commit immediately (no cycle boundaries).
WithEmitTypeIDBudgetPrefix(...)Set the effective type-id prefix used by autogen budget checks
WithRuntimePlannerMode(...)Enable runtime planner mode with no-write-tick semantics, for jobs/tests that drive planning directly from runtime metrics instead of collect-cycle boundaries.

End-to-End Example (Single Flow)

go
// 1) Collector writes metrics.
store := metrix.NewCollectorStore()
meter := store.Write().SnapshotMeter("app")
meter.Counter("requests_total").ObserveTotal(100)

// 2) Engine loads chart template.
engine, err := chartengine.New(
	chartengine.WithEnginePolicy(chartengine.EnginePolicy{
		Autogen: &chartengine.AutogenPolicy{Enabled: false},
	}),
)
// handle err

err = engine.LoadYAML([]byte(`
version: v1
groups:
  - family: App
    metrics: [app.requests_total]
    charts:
      - id: requests
        title: Requests
        context: requests
        units: requests/s
        dimensions:
          - selector: app.requests_total
            name: requests
`), 1)
// handle err

// 3) Prepare plan from flattened+raw reader, emit it, then commit.
// ReadFlatten() is included even for templates with static dimensions
// because it is required for inferred dimensions, structured-family autogen,
// and is the standard pattern.
attempt, err := engine.PreparePlan(store.Read(metrix.ReadRaw(), metrix.ReadFlatten()))
// handle err
plan := attempt.Plan()
defer attempt.Abort()

err = chartemit.ApplyPlan(api, plan, chartemit.EmitEnv{
TypeID:      "plugin.job",
UpdateEvery: 1,
Plugin:      "example",
Module:      "example",
JobName:     "example",
})
// handle err
err = attempt.Commit()
// handle err

PreparePlan Lifecycle

PreparePlan executes a deterministic phase pipeline. Terms like "materialized state" and "route cache" are defined in the Engine State section below.

PhaseSummary
PrepareResolve program/index/cache/materialized state
Validate readerEnsure flattened metadata is available when inferred dimensions are used
ScanIterate series, filter by success sequence and selector, route to chart/dimension accumulators
Cache retainPrune route cache entries not seen in the latest successful sequence
Lifecycle capsEnforce chart/dimension cap policy
MaterializeEmit create/update actions from accumulated state
ExpiryEmit removals for stale charts/dimensions
SortDeterministically sort inferred dimension output

Reader Requirements

ScenarioRequired reader mode
Static named dimensions onlyRead(...) is sufficient (no flatten needed)
Inferred dimensions (name and name_from_label omitted)Must use flattened reader metadata (ReadFlatten)
Structured autogen families (Histogram, Summary, StateSet, MeasureSet)Must use flattened reader metadata (ReadFlatten) or they are not visible
Runtime/default ModuleV2 pathRead(ReadRaw(), ReadFlatten())

If inferred dimensions are present without flattened reader metadata, PreparePlan returns an explicit error.

Action Semantics

ActionMeaning
CreateChartActionMaterialize chart instance (with chart metadata and labels)
CreateDimensionActionMaterialize dimension for a chart
UpdateChartActionEmit chart values for current cycle; unseen dims become IsEmpty=true
RemoveDimensionActionObsolete one dimension
RemoveChartActionObsolete one chart

chartemit normalizes emitted action order by phase:

  1. create chart/dimensions
  2. update values
  3. remove dimensions/charts

Routing and Collision Rules

Each metric series is routed to a chart and dimension based on template selectors. The following rules apply when routing conflicts arise:

RuleBehavior
Template vs autogen chart ID collisionTemplate wins; autogen chart is replaced
Cross-template chart ID collisionExisting owner keeps ownership; subsequent series are silently ignored (see warning below)
Duplicate dimension observations within buildFirst observed dimension metadata wins; values are reduced (summed)

[!WARNING] Cross-template chart ID collisions cause silent data loss — conflicting series are dropped with no error and no log entry. If metrics are missing, check for duplicate rendered chart IDs across template groups.

Lifecycle Defaults and Policy

Default lifecycle policy when template omits lifecycle:

PolicyDefault
max_instances0 (disabled)
expire_after_cycles5
dimensions.max_dims0 (disabled)
dimensions.expire_after_cycles0

Autogen Notes

TopicBehavior
TriggerUnmatched series only when autogen is enabled
Structured familiesAutogen has dedicated source builders for flattened Histogram, Summary, StateSet, and MeasureSet families
Metric metadata usageUses metrix.MetricMeta hints for title/family/unit where allowed
Type ID budgetEnforced via AutogenPolicy.MaxTypeIDLen + effective emit type-id prefix (WithEmitTypeIDBudgetPrefix(...))
LifecycleAutogen applies ExpireAfterSuccessCycles to both chart and dimension expiry (unlike template lifecycle where they default independently)

MeasureSet autogen specifics:

  • chartengine treats MeasureSet as a structured family, similar to StateSet, not as grouped scalar coincidence
  • flattened MeasureSet inputs are expected to carry:
    • SourceKind = MetricKindMeasureSet
    • FlattenRole = FlattenRoleMeasureSetField
    • per-field metric names like <name>_<field>
    • a synthetic reserved field label (measure_field=<field>)
  • the synthetic measure_field label is the authoritative field-identity channel; the per-field metric-name suffix remains for MetricMeta(name) compatibility
  • gauge-like MeasureSet fields autogen with absolute algorithm behavior; counter-like MeasureSet fields autogen with incremental algorithm behavior

Reserved Flattened Label Keys

These label keys are treated specially by chartengine when consuming flattened structured-family or distribution inputs:

Key / PatternMeaning
leHistogram bucket bound label
quantileSummary quantile label
measure_fieldMeasureSet field identity label
<metric-name>StateSet special case: the flattened state name is carried under a synthetic label whose key is the base metric name

Notes:

  • le, quantile, and measure_field are static reserved flattened-label keys in chartengine.
  • StateSet is different: it does not use a global static key; it uses the base metric name itself as the synthetic flattened label key.
  • These keys are part of the flatten contract between metrix and chartengine. Reusing them as ordinary user labels on those flattened inputs is not supported.

Runtime Metrics

chartengine self-instruments to a runtime store by default (disable with WithRuntimeStore(nil)).

FamilyExamples
ChartEngine/Buildbuild success/error/skipped counters, build duration summaries
ChartEngine/Actionsaction counters by kind
ChartEngine/Seriesscanned/matched/filtered series counters
ChartEngine/Route Cachehit/miss/entries/prune counters
ChartEngine/Lifecycleremoval counters by scope/reason
ChartEngine/Plangauges for chart instances/inferred dimensions

Engine State

AreaDesign
ProgramImmutable compiled IR per revision
Engine stateSerialized under Engine.mu for load/build transitions
Route cacheSeries identity + revision keyed cache; retained by successful sequence, pruned on each build
Materialized stateTracks existing chart/dimension instances for incremental create/update/remove decisions; persists across cycles, resets on template reload
DeterminismSorted chart IDs and inferred dimensions provide stable action ordering