metrix

metrix is the metrics storage and read API used by go.d ModuleV2 collectors and runtime/internal components. A collector writes metrics into a store each collect cycle; readers (chart planning, tooling) see a consistent, immutable snapshot of the last successful cycle.

This document is human-oriented. Read it top to bottom as a journey: the plain-language model first, then the collect cycle, then the part that makes metrix safe for unbounded metric streams — descriptor identity and retention — and finally the precise API reference, contracts, and pitfalls.

Audience: ModuleV2 collector authors and framework contributors.

See also: charttpl (template DSL), chartengine (compile + plan), how-to-write-a-collector.md (end-to-end integration).

What metrix Does

A collector does not build charts or hold metric state itself. It observes values into a store, and the store owns everything else: series identity, per-name descriptors, snapshot publication, and lifecycle.

The central rule is simple:

One metric name has exactly one descriptor. The store resolves and publishes that descriptor atomically at commit, keeps it while the name is in use, and evicts it after the name goes idle — so an unbounded stream of metric names (statsd, push, client-driven names) cannot grow the store without bound.

Two store flavors exist for two kinds of producer:

The rest of this document is mostly about CollectorStore (the cycle model and the descriptor lifecycle). RuntimeStore differences are called out where they matter.

Source Map

metrix intentionally keeps its public API and implementation in one Go package. The public import path is github.com/netdata/netdata/go/plugins/pkg/metrix; metrix/selector is the only subpackage and imports the root package. The root package must not import selector, or it would create a cycle.

This single-package shape preserves root-defined public type identity and avoids facade/wrapper overhead on write, commit, read, and Vec hot paths. Use file ownership, tests, and docs to keep the package navigable; introduce a new subpackage only when a future concern has a proven one-way dependency boundary.

Current ownership map:

The Big Picture

A few words carry the whole model. This table is the vocabulary the rest of the document uses; later sections make each part precise.

The flow from a collector write to a reader read:

flowchart LR
    collector("Collector
Observe values") --> staged("Staged frame
this cycle's writes")
    staged --> resolve("Commit: resolve descriptors
fail / supersede / drop")
    resolve --> retain("Apply retention
+ descriptor sweep")
    retain --> publish("Publish snapshot
atomic swap")
    publish --> snap("Immutable snapshot")
    snap --> reader("Readers
chart planning / tooling")

    abort("AbortCycle
scrape failed") -. discard staged .-> staged

    classDef input fill:#eef1f4,stroke:#8b949e,color:#24292f;
    classDef sched fill:#ece2ff,stroke:#8250df,color:#3b1f6b;
    classDef commit fill:#d8f3e0,stroke:#2da44e,color:#0b3d1f;
    classDef hard fill:#ffdcdc,stroke:#cf222e,color:#5a0b0b;

    class collector,reader input;
    class staged,resolve,retain sched;
    class publish,snap commit;
    class abort hard;

Key properties that fall out of this shape:

• Reads do not take the commit lock. A reader holds an immutable snapshot; commits swap in a new one. The first index or flattened-view request for a CollectorStore snapshot synchronizes one builder; subsequent readers use the published immutable result concurrently.
• Writes are invisible until commit. A half-collected or failed cycle never leaks partial data to readers.
• Commit is transactional. Either the whole cycle publishes, or (on AbortCycle / an unresolvable conflict) nothing changes.

The Collect Cycle

A CollectorStore write only exists inside a cycle. The job runtime drives the cycle around the collector's Collect; ModuleV2 collectors just write.

RuntimeStore has no cycle API: writes commit immediately, each producing a new overlay snapshot, and it is stateful-only (snapshot-mode registration returns an error; calling snapshot record methods panics).

Metric Identity: Names, Series, and Descriptors

Every write resolves to a series — the metric name plus a canonical key built from its sorted labels. Many series can share one name (same metric, different label values).

All series of a name share one descriptor. The descriptor is the name's contract and is the thing whose growth metrix bounds:

• Authority — kind, mode, freshness, window, and the type schema (histogram bounds / summary quantiles). Two writes with different authority for one name are a conflict.
• Declaration — metadata (description, unit, chart family/priority, float). Compatible declarations for one authority are merged into a single canonical descriptor; a genuine conflict (e.g. two different units) fails.

At commit the store computes exactly one canonical descriptor per accepted name and stamps every surviving series of that name with it, so readers, the registry, and every series agree regardless of write or label order.

Descriptor Lifecycle and Retention

This is the core of why metrix is safe for push-style, client-driven metric names. It applies to CollectorStore (RuntimeStore has its own bounded retention path).

Two coupled retentions

Two things age out, on the successful-commit clock (only successful commits advance it; aborted cycles do not):

1. Series retention — a series not observed for expireAfterSuccessCycles successful commits is evicted. maxSeries optionally caps the total series count, evicting the oldest first.
2. Descriptor grace — after a name's last series is gone, its descriptor is kept for descriptorGraceCycles more successful commits (in case the name returns), then swept from the registry.

So a fully idle name's descriptor lives expire + grace successful commits past its last observation. Defaults: expire = 10, maxSeries = 0 (disabled), grace = expire (so grace ≥ expire holds by construction).

The descriptor state machine

flowchart LR
    new("New name observed") --> live

    live("LIVE
has ≥1 live series")
    grace("ZERO-LIVE-IN-GRACE
no series, within grace")
    evicted("EVICTED
removed from the registry")

    live -->|last series evicted by retention| grace
    grace -->|observed again| live
    grace -->|grace elapsed → universe sweep| evicted
    evicted -->|name reappears| live

    classDef commit fill:#d8f3e0,stroke:#2da44e,color:#0b3d1f;
    classDef effect fill:#ffe6cc,stroke:#e36209,color:#5a2a00;
    classDef hard fill:#ffdcdc,stroke:#cf222e,color:#5a0b0b;
    classDef input fill:#eef1f4,stroke:#8b949e,color:#24292f;

    class new input;
    class live commit;
    class grace effect;
    class evicted hard;

• LIVE — at least one series exists. The descriptor is authoritative; a conflicting re-registration of a truly-live name fails loud.
• ZERO-LIVE-IN-GRACE — the series aged out but the descriptor lingers, so a briefly-absent name resumes without re-registration churn.
• EVICTED — swept from the registry. The name is a clean slate: it may reappear with the same or a changed contract and re-register without error.

The sweep runs at commit, after retention, over the whole descriptor universe (the names in the registry) in one pass — bounded work, no per-series refcounts.

Conflict resolution at commit

Because registration is staged and resolved at commit (never a mid-cycle panic for collector code), a name observed with an incompatible authority is resolved by state:

Consumers that cache per-name state

A consumer that caches handles keyed off a metric name (the prometheus writer) must keep its cache alive at least as long as metrix keeps the descriptor, or it would re-register a name metrix still holds. The store exposes this via an optional interface (type assertion; not part of CollectorStore, so existing implementations and fakes keep compiling):

type DescriptorRetention interface {
    DescriptorRetentionWindow() uint64 // expire+grace; DescriptorRetentionUnbounded if expiry is disabled
    SuccessfulCommits() uint64         // the successful-commit clock the window is measured in
}

DescriptorRetentionWindow() returns DescriptorRetentionUnbounded (math.MaxUint64) when series expiry is disabled (WithExpireAfterSuccessCycles(0)), meaning descriptors can live indefinitely — a consumer must then never age out its cached state for that name.

Reading

Read(...) returns an immutable Reader over the latest published snapshot. Two independent axes control what and how it reads:

• Raw (ReadRaw()) — bypass freshness filtering; return all committed series regardless of when last observed.
• Flatten (ReadFlatten()) — project complex types (Histogram, Summary, StateSet, MeasureSet) into individual scalar series.

Freshness decides visibility for non-raw reads:

• FreshnessCycle — the series must have been observed in the latest successful cycle to be visible.
• FreshnessCommitted — the series stays visible as long as it is committed, even if not re-observed this cycle.

Window decides how stateful histogram/summary instruments accumulate:

• WindowCumulative — observations accumulate across cycles.
• WindowCycle — observations reset each cycle.

The Reader interface exposes typed getters (Value/Delta/Histogram/Summary/StateSet/MeasureSet), metadata (SeriesMeta/MetricMeta/CollectMeta), host scopes, and iteration helpers (ForEachByName/ForEachSeries/ForEachMatch).

Snapshot-owned projection and scope indexes

CollectorStore owns read acceleration at the published-snapshot boundary:

• The first Read(ReadFlatten()) for one exact snapshot builds the complete flattened scalar projection and its deterministic indexes.
• Later flattened readers of that snapshot reuse the same immutable projection.
• A successful commit and a failed/aborted attempt each publish a new exact snapshot, so freshness metadata and cached projections cannot cross the boundary.
• Scalar gauge/counter series are shared with the canonical snapshot because committed series are immutable. Structured families allocate their synthetic flattened series once per snapshot.
• Raw and flattened readers use immutable per-scope name indexes. Scoped lookup and iteration therefore depend on that scope's series, not on peer-scope cardinality.

RuntimeStore deliberately keeps per-read flattening because its immediate-write overlay snapshots have a different lifetime and reuse profile. It still uses the same deterministic reader indexes within each read.

Host-scope reads and metadata

• Read() selects the default host scope; ReadHostScope(key) selects one explicit scope.
• HostScopes() returns every scope present in the snapshot, independent of the reader's active scope. Returned metadata is defensively cloned.
• MetricMeta(name) is scope-local. Fresh readers prefer visible series and may fall back to stale metadata only inside the active scope; raw readers may use stale metadata in that scope. Neither mode falls back to a peer scope.
• CollectorStore readers optionally implement FreshVisibleHostScopesReader. Its FreshVisibleHostScopes() result contains only scopes with at least one series visible under normal freshness rules and is independent of ReadRaw().

Instruments

An instrument is declared on a meter and identified by its name; observing through it writes a series.

Kinds: Gauge, Counter, Histogram, Summary, StateSet, and MeasureSet (below). Instruments accept options:

MeasureSet

MeasureSet stores one logical metric family with a fixed ordered list of named numeric fields — a structured family, similar to StateSet for chart autogen.

• Family-level semantics — a family is either gauge-like or counter-like; never mixed per field.
• Family-level metadata — Description, ChartFamily, and Unit apply to the whole family.
• Field-level schema — MeasureFieldSpec declares per-field Name and Float.

Writers

Write contract

• Prefer named write helpers over raw positional MeasureSetPoint values.
• Snapshot handles support full-family positional and named writes.
• Stateful handles additionally support singular field writes (SetField/AddField), which update only the addressed field on top of the committed/staged family.
• Snapshot singular field writes do not exist (phase 1): snapshot mode models one sampled family point per cycle; partial field visibility is deferred.
• Named full-family writes require the exact declared field set — missing or unknown fields panic.

Schema example

store := metrix.NewCollectorStore()
meter := store.Write().SnapshotMeter("svc")
latency := meter.MeasureSetGauge(
	"latency",
	metrix.WithMeasureSetFields(
		metrix.MeasureFieldSpec{Name: "value"},
		metrix.MeasureFieldSpec{Name: "ratio", Float: true},
	),
	metrix.WithUnit("seconds"),
)
latency.ObserveFields(map[string]metrix.SampleValue{
	"value": 1.5,
	"ratio": 0.5,
})

Stateful singular-write example

store := metrix.NewRuntimeStore()
meter := store.Write().StatefulMeter("svc")
usage := meter.MeasureSetGauge(
	"usage",
	metrix.WithMeasureSetFields(
		metrix.MeasureFieldSpec{Name: "value"},
		metrix.MeasureFieldSpec{Name: "limit"},
	),
)

usage.SetFields(map[string]metrix.SampleValue{"value": 10, "limit": 20})
usage.SetField("value", 15)
usage.AddField("limit", 3)
// committed point == metrix.MeasureSetPoint{Values: []metrix.SampleValue{15, 23}}

Flattened View Mapping

Read(ReadFlatten()) projects non-scalar families into scalar series:

Histogram bucket flattening emits non-overlapping range bucket totals while keeping the le label as the bucket upper bound: the first finite bucket is <= le[0]; each later finite bucket is (previous le, current le]; +Inf is count - last finite cumulative bucket. Typed Histogram() reads stay canonical cumulative points.

Flatten metadata is exposed via SeriesMeta.Kind, SeriesMeta.SourceKind, and SeriesMeta.FlattenRole. MeasureSet flattening keeps per-field metric names for MetricMeta(name) compatibility and adds a synthetic measure_field=<field> label for explicit field identity.

Configuration

NewCollectorStore(opts ...CollectorStoreOption) is variadic and additive; existing callers keep working. The options tune the retention model above:

Eviction and drop activity is exposed as cumulative counters on the per-snapshot CollectMeta (read via Reader.CollectMeta()); metrix itself never logs:

Usage Snippets

Collector write path

store := metrix.NewCollectorStore()
meter := store.Write().SnapshotMeter("mysql")
qps := meter.Counter("queries_total")
qps.ObserveTotal(42)

Read path for planning

reader := store.Read(metrix.ReadRaw(), metrix.ReadFlatten())
value, ok := reader.Value("mysql.queries_total", nil)
_ = value
_ = ok

Direct MeasureSet read

reader := store.Read()
point, ok := reader.MeasureSet("svc.latency", nil)
_ = point
_ = ok

For a complete collector integration pattern (cycle management, error handling), see how-to-write-a-collector.md.

Contracts and Pitfalls

• Descriptor lifetime is not permanent. A fully idle name evicts after expire + grace successful commits and can then be re-registered with a changed contract. Consumers that cache per-name handles must read DescriptorRetention and not outlive the window.
• Aborted cycles freeze the clocks. Retention, grace, and the sweep advance only on successful commits, so endpoint downtime does not age anything out.
• Label sets — LabelSet is store-owned; do not share between stores.
• Counter deltas — Delta() requires a contiguous sequence (N, N+1). In CollectorStore this is per-cycle (a missed successful cycle breaks the delta); in RuntimeStore it is per-series per-write (skipping a write does not break it).
• Snapshot freshness — snapshot-mode instruments cannot use FreshnessCommitted.
• Runtime writes — RuntimeStore rejects snapshot-mode registration with an error; calling snapshot record methods (ObserveTotal, ObservePoint) panics. MeasureSet families work only through StatefulMeter(...).
• MeasureSet named writes require the exact declared field set; snapshot singular field writes are absent in phase 1.
• MeasureSet counter semantics — stateful counter-like families reject negative AddPoint(...) deltas, like scalar counters.
• Window/freshness coupling — stateful histogram/summary with WindowCycle requires (and silently forces) FreshnessCycle; an explicit non-Cycle freshness with WindowCycle is an error.
• Truly-live schema conflicts fail the commit. Re-registering an actively-observed name with a different kind/mode/schema fails loud (transactional); an idle name is superseded instead. Init-time registration keeps a synchronous panic.
• Summary NaN quantiles — a summary point may carry NaN quantile values (e.g. an empty window); they are stored (only Inf is rejected) and render as a chart gap downstream. Count and Sum must be finite.

Internal Architecture Notes