.agents/skills/project-writing-collectors/SKILL.md
You are about to add or modify data collection in the Netdata Agent. This skill is a manifesto and a routing map. It tells you the mindset to apply, the principles you cannot violate, the ways the dashboard gets shaped from upstream data, the quality bar that separates a draft from a shippable collector, and where to look for depth. It is not a tutorial — the deep references already exist in the repo. Your job is to know they exist, pick the right one, and produce work that blends with the patterns the maintainers already accept.
The skill is organized as: AI fast path → mental model → best practices → dashboard shaping → quality bar → environment reference → applied per data type → applied per domain. For go.d work, follow the fast path first; for other collector families, read top to bottom on your first pass and come back to specific sections as the task narrows.
For implementation agents, route to the concrete workflow first and use the rest of this skill as background:
src/go/AGENTS.md, then
src/go/plugin/go.d/docs/how-to-write-a-collector.md,
.agents/skills/project-writing-go-modules-framework-v2/SKILL.md, and
.agents/skills/integrations-lifecycle/recipes/add-go-collector.md.src/go/AGENTS.md, the collector's
local files, .agents/skills/integrations-lifecycle/consistency.md, and
.agents/skills/integrations-lifecycle/recipes/update-collector.md.src/go/AGENTS.md,
src/go/plugin/go.d/docs/migrate-v1-to-v2.md, and the V2 skill before
changing code.src/go/plugin/framework/docs/changing-framework-code.md before writing
code..agents/skills/project-prometheus-profiles/SKILL.md and every source it marks as
mandatory, then run the repository real-pipeline validator it documents.Do not use this broad skill as the only implementation guide for go.d work.
How to think about Netdata data collection. Internalize this before designing anything.
The Agent ships on >1.5M new daily installs across physical servers, VMs, containers, IoT devices, embedded systems, and exotic Unixes. Default collection is 1-second; many collectors raise it (ping 5s, SNMP 10s) when the source warrants it. Anything you do inside the collection cycle — allocate, log, reconnect, retry, parse, format — is multiplied by that population. Hot-path discipline is the entry ticket, not an optimization.
How dimensions group into charts and how labels attach to instances is the dashboard the user sees. Mirroring upstream data structures one-to-one produces a chart per metric, which is unusable. NIDL — Nodes, Instances, Dimensions, Labels — is the model. Every dashboard-shaping mechanism (§3) feeds into it.
Chart context, chart IDs, dimension IDs, instance labels — once shipped, they bind health alerts, dashboards, exports, anomaly detection, ML jobs, streaming consumers, and Netdata Cloud. Renaming silently breaks all of them. Treat them as permanent.
When you cannot measure a value this iteration, emit nothing for that dimension. The dashboard renders the gap; the user knows collection is broken. Defaulting to 0 fabricates a working state and hides the bug. Past pain in src/collectors/proc.plugin/proc_net_dev.c (search shouldn't use 0 value, but NULL).
When the collector knows an entity has gone away — a process exited, a container was removed, a profile target was dropped, a network interface disappeared, a managed device went offline — mark its chart obsolete. The dashboard then renders it as historical, not as actively collected; alerts stop binding to it; streaming and ML stop costing for it.
This is a truthfulness principle, not a cardinality one. It applies at any cardinality, including a single instance. Without obsoletion, the chart looks alive on the dashboard, alerts may continue evaluating against frozen data, and the user is misled about what is and isn't being collected.
Mechanics:
rrdset_is_obsolete___safe_from_collector_thread() in src/database/rrdset.c:116 flags RRDSET_FLAG_OBSOLETE. Reverse with rrdset_isnot_obsolete() (line 140) when the entity reappears.c.Obsolete = true or MarkRemove() on the chart marks it obsolete. go.d V2: chart lifetime is controlled by charts.yaml lifecycle policy and chartengine; start from src/go/plugin/go.d/docs/how-to-write-a-collector.md for new collectors and src/go/plugin/go.d/docs/migrate-v1-to-v2.md for migrations.Specs, vendor protocols, RFCs, and SDK behavior move. Before you design a collector or interpret a payload:
Prior-knowledge mistakes that recur: confused field names in NetFlow v5 vs v9 vs IPFIX, wrong endianness on a vendor MIB, outdated PostgreSQL pg_stat_* columns, deprecated Kubernetes API resources.
Specs leave many decisions implementation-defined. Vendor implementations bend specs in well-known ways. When you face an interpretation dilemma:
This is how you avoid shipping a parser that fails on the first real device. If you have a local mirror of monitoring projects, use it; otherwise clone the relevant upstreams to /tmp/ and read their source.
The repo holds many go.d modules and internal C plugins. Maintainer patterns live there, not in any prose doc. After you've reality-checked the upstream protocol, pick the closest existing Netdata collector by domain and mirror its structure. New go.d modules MUST use framework V2 and start from the current V2 authoring guide — see §5.3.
When one collector talks to N targets (SNMP devices, remote DBs, cloud APIs, IPMI hosts, vCenter clusters), each target is a vnode so its metrics, alerts, and RBAC behave as if it were a separate node in Netdata Cloud. Every remote-target collector wires vnodes from the start.
For Go v2 collectors that route one job's samples to multiple virtual nodes, use first-class metrix.HostScope rather than adding vnode identity as normal metric labels. Write per-resource metrics through scoped meters or vecs such as meter.WithHostScope(scope), and leave metrics unscoped when they should follow the default job vnode or global host path. Scope keys must be stable for the virtual node identity; unbounded scope cardinality has the same operational cost profile as unbounded chart/cardinality growth.
Design for usefulness, not raw count. Bound cardinality (§2.5), and never ship "one chart per request / per PID / per ephemeral connection" without bounds.
Per-job source priority: stock < discovered < user < dyncfg, matched by job identity. A higher-priority source replaces a lower-priority job with the same identity; non-colliding jobs continue to load. IaC users configure via files in /etc/netdata; dashboard users configure via DYNCFG; both paths must work for the same collector.
Framework-agnostic, ordered by impact.
You MUST aim for the clean end state, not the smallest diff. While implementing, keep checking whether the design still looks like the structure maintainers should want after the work is complete.
At each coherent batch, you MUST check for scope drift. If the work exposes an independent collector cleanup, framework change, docs correction, or migration, either defer it explicitly or submit it as its own step before continuing.
Source test data based on what you're collecting:
src/crates/netflow-plugin/testdata/flows/ with sourcing recorded in testdata/ATTRIBUTION.md — do the same for any new fixtures with redistribution-sensitive provenance.Don't fabricate test data the parser passes by accident. Don't skip tests "because this protocol can't be tested locally" — that's exactly when fixtures matter most. Standard go.d test-function names: Test_testDataIsValid, TestCollector_ConfigurationSerialize, TestCollector_Init, TestCollector_Check, TestCollector_Collect — match the convention in adjacent collectors. Functions get a dedicated validator at src/go/tools/functions-validation/ (E2E plus schema checks).
For Go tests, prefer table-driven tests using map[string]struct{} keyed by
test-case name when cases share setup and assertion shape. Use separate test
functions only when setup or assertions are materially different. Prefer map
keys over a name field in []struct{} so case names stay prominent and
order-independent.
Collect() runs every update_every seconds. It MUST:
Init() / New() and reuse them. Reset at the top of
Collect() only when needed; see cato_networks/metrix.go for the typed
V2 metric-instrument pattern.Anti-pattern (search and avoid): mx := make(map[string]int64) per Collect() (e.g., src/go/plugin/go.d/collector/ap/collect.go). Don't allocate fresh structures per cycle. Don't reconnect every cycle.
Every error log answers three questions: what operation, what target, what was expected vs observed. Wrap errors with context (Go: fmt.Errorf("...: %w", err)); preserve the cause; check return codes from system calls and library functions.
Don't return a bare err with no context. Don't log "failed". Don't ignore syscall returns or library NULLs.
debug inside the collection loop.warn or error once per known-recoverable condition, gated by an internal flag — never per cycle.info / notice for once-at-startup events.error severity for operator-actionable issues; transient conditions are warn.Past pain: an ebpf.plugin regression flooded logs because the collection loop logged every PID allocation. Per-cycle logs are forbidden.
When a collector emits one chart per discovered entity (process, connection, profile target, container, schema, queue, route), bound the count and let the operator scope it. (Obsoletion of entities the collector knows have gone is a separate concern — see §1.5.)
max_* is REQUIRED for entities that may grow without bounds. Without a cap, a single misbehaving target (a runaway log rotator, a container churn loop, a vendor-specific deep table) can produce thousands of charts.
max_* MUST be coupled with selectors. A cap alone silently truncates whatever happens to land in the first N entries — the operator has no say in which entities survive. A selector lets the operator pick what's actually important. Cap and selector together: cap protects the system, selector lets the operator drive.
Where to filter — depends on what the monitored application exposes:
max_* and adds an aggregated "Other" chart that sums whatever was capped. Don't silently drop — totals must remain truthful even when individual instances are hidden.Anti-patterns:
For go.d V2 collectors, keep selector/cap behavior in the collector design and
document the public config only when the operator has a real decision to make.
Start from src/go/plugin/go.d/docs/how-to-write-a-collector.md.
Public tunables are part of the collector consistency contract. When a config
option is added, removed, renamed, or given a new default, you MUST follow
.agents/skills/integrations-lifecycle/consistency.md; you MUST NOT update
only the Go struct or only the docs. The stock .conf shows safe,
representative examples -- not necessarily every tunable.
Collectors MUST NOT hardcode timeouts, paths, ports, or credentials. Stock config and schema MUST NOT contradict each other.
config_schema.json is a form contract, not a validation layer. Nothing in
the agent validates a job config against it; runtime enforcement is the
collector's own validate(). Write it for the operator filling the form:
minItems. The dyncfg form
materializes such an array even when the operator never opened that section,
fills its required leaves with nulls, and then refuses to delete the item it
created — producing a job that can never be saved. minItems belongs only on
an array the parent required list demands, or one carrying a default.
This rule is enforced for every collector by
src/go/plugin/go.d/collector/config_schema_test.go
(TestConfigSchemasDoNotMaterializeOptionalArrays). Express "non-empty when
present" in validate() instead.dependencies branch MUST NOT also be declared as a
plain sibling property. The branch exists so the form shows that key only for
the selected discriminator value; declaring it twice makes the form show it
for every value, and leaves the runtime deciding what a field from an
unselected branch means. Enforced for every collector by
src/go/plugin/go.d/collector/config_schema_test.go
(TestConfigSchemasDoNotDeclareBranchKeysAsProperties).["array", "null"] / ["object", "null"]. A YAML key written with no value
decodes to nil, which the runtime reads as "absent" and accepts, so a bare
"array" makes the schema reject a config that works. Keep the scalar type on
required lists and on item schemas, where a null really is invalid. This union
is the majority convention in the tree.add. Keep
both layers deliberate — but note a deliberate asymmetry is fine where only one
layer can act on it: minimum: 1 on an option whose code treats 0 as "use the
default" is correct, because the form should not offer 0 while a file may
legitimately contain it.Do NOT write tests that describe what a schema file contains — its required
list, its defaults, its ui:help prose, its placeholders. The schema is
hand-authored data, so such a test restates the file it reads: when the schema is
wrong the test pins the mistake as correct and fixing it means editing the test.
Only two kinds of schema assertion carry weight:
minItems check, or a per-collector check that a field whose
omission drives runtime precedence carries no default (see
cloudwatch/config_test.go, TestConfigSchema_FormContract).Everything else about config behavior belongs in tests of validate(), which is
the layer that actually runs. Feed every config test the collector's own Config
type — mutate a valid base config per case:
map[string]any payloads. That is a transport detail outside
the collector, and round-tripping one through yaml.Marshal fabricates a
pipeline production never runs.Config changes while testing yaml.v3 rather than
collector code. "Unknown keys are ignored" and "a mapping fails where a sequence
is expected" are decoder behavior, and a guard against a spelling that never
shipped protects nothing.testdata/config.json and
testdata/config.yaml, driven by collecttest.TestConfigurationSerialize from
TestCollector_ConfigurationSerialize. Put a representative value in those
fixtures — including any explicit zero whose round-trip carries meaning, such as
a 0s duration that must not be read as "inherit" — instead of writing a
collector-local decode test.Config field encodes a contract, test the accessor that
reads it rather than the decoding that filled it. cloudwatch/config_test.go,
TestRuleConfig_effectiveResourceTagFilters covers the pointer-to-slice
nil-versus-empty rule that way.internal/awsauth), and
compile-stage reference rules belong with the compile tests, not the config
tests.Put every default in the constructor. New() MUST set a value for each option
that has one, so it returns a usable collector — callers and tests rely on that,
and a constructor that yields an unusable object is a footgun. Then:
applyDefaults, run from Init) handles only what the
constructor cannot see: an explicit sentinel such as a 0 the operator typed,
or a nil pointer on a config not built through New().if v <= 0 { v = default } at the point
of use, or an accessor that re-applies the default). It can never fire once the
constructor and normalizer have run, so it is dead code that still has to be
read, and it hides which layer owns the value. Read the field directly.Credentials use the ${env:}/${file:}/${cmd:}/${store:} indirection — see src/collectors/SECRETS.md. Privileged operations route through src/collectors/utils/ndsudo.c.
Several artifacts are produced from upstream definitions and MUST NOT be hand-edited:
integrations/<name>.md — generated from metadata.yaml (banner: DO NOT EDIT THIS FILE DIRECTLY).ibm.d modules — generated README.md, metadata.yaml, config.go, zz_generated_*.go from contexts.yaml via go generate.charts-derive proc-macro.When a generated file looks wrong, fix the source of truth (metadata.yaml, contexts.yaml, derive macro input) and regenerate. Note: go.d uses //go:embed for static assets — there is no go generate step.
Collector consistency has one detailed checklist:
.agents/skills/integrations-lifecycle/consistency.md. Treat code,
integration metadata, taxonomy, config, stock examples, alerts, and generated
documentation as one unit, but do not maintain a second artifact matrix here.
If a collector exposes a Function, its response shape MUST also conform to the
relevant Function schema, such as src/plugins.d/FUNCTION_UI_SCHEMA.json or
src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json.
When one collector needs data from another, use netipc — never shell out, open private sockets, poll log files, or reinvent IPC. In-tree libraries:
src/libnetdata/netipc/src/go/pkg/netipc/src/crates/netipc/Both clients (consume) and servers (offer) exist in all three languages. Real example: src/collectors/cgroups.plugin/cgroup-netipc.c is a netipc server offering cgroup metadata to other plugins. Upstream spec, tests, fuzz suite: https://github.com/netdata/plugin-ipc.
Set Vnode in job config when the collector has one remote target. For Go V2
collectors that emit multiple remote nodes from one job, use
metrix.HostScope; see .agents/skills/project-writing-go-modules-framework-v2/go-v2-host-scope.md and
src/go/plugin/go.d/docs/how-to-write-a-collector.md. Past pain: an older
refactor had to retroactively split job-name validation per vnode/domain because
earlier collectors had not accounted for it.
The dashboard is built from charts. The way upstream data turns into charts depends on the ingestion path. Six mechanisms exist; pick the one that matches your collector and learn how it shapes the result.
Nodes, Instances, Dimensions, Labels. This is the conceptual model every other mechanism feeds into. Read docs/NIDL-Framework.md before designing metrics. Group dimensions into charts that answer one operational question. Use labels for instance and context annotations. Pick the right chart type (line, area, stacked, heatmap — see src/database/rrdset-type.h) and dimension algorithm (absolute, incremental, percentage-of-incremental-row, percentage-of-absolute-row — see src/database/rrd-algorithm.h, documented in src/plugins.d/README.md).
Common bugs: absolute on a counter (counters are incremental); line when stacked is the right shape (CPU states, disk-time breakdown). Reuse shared metric definitions from src/collectors/common-contexts/ for C plugins.
SNMP collection is profile-driven. A profile is a YAML document declaring OIDs, metric definitions, table indexing, units, chart families, and selectors. Stock profiles ship from src/go/plugin/go.d/config/go.d/snmp.profiles/default/; spec at src/go/plugin/go.d/collector/snmp/profile-format.md (~2000 lines).
Adding or extending SNMP coverage means writing or extending a profile, not adding code. The SNMP topology collector (snmp_topology) builds on top of profiles — extending profiles is usually the right starting point for topology work too.
Past pain: pre-profile SNMP code required per-vendor branches that became unmaintainable. Don't hardcode OID-to-metric mappings inside a custom collector or vendor branch.
synthetic_charts — operator-curated dashboardsThe statsd plugin lets the operator group raw statsd metrics into curated charts via INI configs at /etc/netdata/statsd.d/*.conf. Each config defines:
[app] — match raw metrics by pattern, group them under an application name[dictionary] — rename raw metric names to display namestitle, family, context, units, type, and explicit dimension = lines mapping source metrics to display dimensionsWildcard patterns extract dimension names from the matched portion: dimension = pattern 'myapp.api.*.200' '' last 1 1 creates dimensions named after the wildcard match. Three-layer dimension lookup (dimension name in dictionary → metric name in dictionary → fallback to original). Stock examples: src/collectors/statsd.plugin/k6.conf, src/collectors/statsd.plugin/asterisk.conf. Full spec: src/collectors/statsd.plugin/README.md lines 397-639.
This is the most operator-controllable shaping mechanism — the dashboard is whatever the operator declares.
Netdata's OTEL plugin (src/crates/otel-plugin/) accepts any OTLP gRPC metric. Mapping is generic by default — all resource attributes, scope attributes, and data point attributes become chart labels — but the operator controls routing via per-metric YAML files at /etc/netdata/otel.d/v1/metrics/*.yaml. Key knobs:
instrumentation_scope.name / version — regex match to scope an entry to a specific OTel instrumentationdimension_attribute_key — which data point attribute becomes the dimension name (default: "value"); other attributes become chart labelsinterval_secs, grace_period_secs — per-metric timing overridesAggregation temporality drives the chart algorithm: Gauge → absolute, Sum delta → DeltaSum, Sum cumulative monotonic → CumulativeSum, Sum cumulative non-monotonic → treated as Gauge (src/crates/otel-ingestor/src/chart.rs).
The plugin does not recognize OTel semantic conventions specifically (host.name, service.name, deployment.environment) — they pass through as labels. Cardinality control is metrics.max_new_charts_per_request in otel.yaml. Stock examples: src/crates/otel-ingestor/configs/otel.d/v1/metrics/.
The generic Prometheus scraper (src/go/plugin/go.d/collector/prometheus/) auto-maps from the exposition format:
counter, gauge, histogram, summary) → chart type and dimension algorithm_sum, and _count charts; summaries always produce _sum and _count, plus a
quantile chart when quantiles exist_total (counter), _bucket + le label (histogram), _sum, _count, quantile label (summary), _info (skipped)_seconds, _bytes, _hertzOperator controls (profiles documented in src/go/plugin/go.d/collector/prometheus/profile-format.md, relabeling in
src/go/plugin/go.d/collector/prometheus/relabel/README.md):
selector job option (allow/deny on metric name and label values, syntax in
src/go/pkg/prometheus/selector/README.md) and fallback_type glob patterns for untyped metrics.relabeling is operator policy and runs before profile selection. Profile-root relabeling
uses the same Prometheus-compatible block/rule format after selection and owns stable exporter normalization required
by that profile's charts. Do not duplicate profile-required normalization as an optional job recipe.selector -> job relabeling/safety -> fallback type + profile selection -> selected profile relabeling/safety -> final gates -> charts. A profile cannot normalize itself into
selection. Untyped classification is bound before profile relabeling, so a final rename cannot create or change it.auto, configured entry order in exact, and configured
entries followed by the remaining name-ordered auto profiles in combined. Root match and block matchers classify
original source names; rule results and output names do not affect dispatch. All selected templates consume the final
names and labels, so authors must account for cross-profile interactions.match/app/relabeling/autogen.selector/template YAMLs, stock under
src/go/plugin/go.d/config/go.d/prometheus.profiles/default/, user under
/etc/netdata/go.d/prometheus.profiles/) ship curated per-exporter dashboards — the Prometheus analog of
statsd synthetic_charts. Metrics not covered by an authored profile chart keep their autogen charts unless an
applicable profile selector rejects them. Each selector is limited to its profile's match scope; when scopes
overlap, every applicable selector must accept the series. This changes fallback charts only; use the job selector
or a relabeling drop action to discard samples.Use .agents/skills/project-prometheus-profiles/SKILL.md when creating or materially
reviewing a profile. It teaches the dashboard-design reasoning and the
real-pipeline validation boundary; schema validity alone is not semantic
dashboard approval.
Chart priorities (priority field in C, Priority in Go) drive UI ordering. C plugins follow conventions in src/collectors/all.h. Don't pick priorities arbitrarily; mirror an adjacent collector's range.
A collector is production-quality when it satisfies all of:
Cleanup() cycles or job reloads.Collect() finishes well under one cycle even on a slow target.max_* and selectors so the operator can scope them.incremental for counters, etc.)?.agents/skills/integrations-lifecycle/consistency.md, including the rule that generated integration pages are not hand-authored sources?profile-format.md), and should that profile suppress unmatched fallback
charts with a scoped autogen.selector while retaining their samples?go generate after touching contexts.yaml?collector/init.go import, go.d.conf, stock conf, README)?max_* + selectors? Aggregated "Other" bucket or upstream-supplied aggregation present where applicable?Reference section. Use it after the mental model and best practices have framed your task.
| Family | Lang | Platforms | Where in repo | Scope |
|---|---|---|---|---|
proc.plugin | C | Linux | src/collectors/proc.plugin/ | Kernel /proc and /sys |
apps.plugin | C | Linux/FreeBSD/macOS/Windows | src/collectors/apps.plugin/ | Per-process and per-user/group; processes Function |
cgroups.plugin | C | Linux | src/collectors/cgroups.plugin/ | Containers and control groups |
ebpf.plugin | C + eBPF | Linux | src/collectors/ebpf.plugin/ | Kernel function tracing |
network-viewer.plugin | C | Linux | src/collectors/network-viewer.plugin/ | L3/L4 sockets; topology: Functions |
systemd-journal.plugin / windows-events.plugin | C | Linux/Windows | src/collectors/{systemd-journal,windows-events}.plugin/ | Log/event explorers via Functions |
systemd-units.plugin | C | Linux | src/collectors/systemd-units.plugin/ | systemd unit state |
windows.plugin | C | Windows | src/collectors/windows.plugin/ | Windows performance counters |
freebsd.plugin / macos.plugin | C | platform-specific | src/collectors/{freebsd,macos}.plugin/ | OS analogs of proc.plugin |
statsd.plugin | C | All | src/collectors/statsd.plugin/ | StatsD ingestion + synthetic_charts |
log2journal | C | Linux | src/collectors/log2journal/ | Parse application logs into the systemd journal |
| Niche C plugins | C | various | src/collectors/<name>.plugin/ | freeipmi, nfacct, tc, xenstat, debugfs, diskspace, slabinfo, idlejitter, timex, cups, ioping, perf |
go.d.plugin | Go (no CGO) | All | src/go/plugin/go.d/ | Application integrations |
ibm.d.plugin | Go + CGO | Linux, IBM i | src/go/plugin/ibm.d/modules/ | IBM workloads (DB2, IBM i / AS-400, IBM MQ, WebSphere) |
netflow-plugin | Rust | Linux | src/crates/netflow-plugin/ | NetFlow v5/v9, IPFIX, sFlow |
otel-plugin | Rust | Linux | src/crates/otel-plugin/ | OpenTelemetry metrics + logs ingestion (logs queryable via the otel-logs Function) |
charts.d.plugin / python.d.plugin | Bash / Python | All | src/collectors/{charts,python}.d.plugin/ | Legacy — do not add new modules |
Path conventions: internal C plugins → src/collectors/<name>.plugin/; Go orchestrators → src/go/plugin/{go.d,ibm.d}/; Rust plugins → src/crates/<name>/.
| If you are doing… | Start with |
|---|---|
| New off-the-shelf application integration (no CGO) | src/go/plugin/go.d/docs/how-to-write-a-collector.md; primary V2 reference: src/go/plugin/go.d/collector/cato_networks/ |
| Migrating existing go.d collector to V2 | src/go/plugin/go.d/docs/migrate-v1-to-v2.md; V2 mechanics: .agents/skills/project-writing-go-modules-framework-v2/SKILL.md |
| New IBM workload integration (CGO) | src/go/plugin/ibm.d/AGENTS.md, src/go/plugin/ibm.d/framework/README.md |
| New Rust plugin | SDK at src/crates/netdata-plugin/; reference: src/crates/netflow-plugin/ |
| New SNMP profile (no code change) | src/go/plugin/go.d/collector/snmp/profile-format.md |
| New interactive Function | src/go/plugin/framework/functions/README.md, src/plugins.d/FUNCTION_UI_SCHEMA.json, src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md |
| Topology work | .agents/skills/project-create-topology/SKILL.md, src/go/pkg/topology/v1, src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json |
| Auto-discovery for a new go.d module | rules under src/go/plugin/go.d/config/go.d/sd/; engine: src/go/plugin/agent/discovery/ |
| OTEL ingestion | src/crates/otel-plugin/ (ingest logic in src/crates/otel-ingestor/) |
| Log ingestion (parse → journal) | src/collectors/log2journal/ and log2journal.d/ rules |
| New external plugin in any language | src/plugins.d/README.md (PLUGINSD protocol) |
| New internal C plugin | src/collectors/README.md; mirror an adjacent collector |
| Cross-plugin data enrichment | netipc libraries (§5.4) |
| Privileged operations | src/collectors/utils/ndsudo.c |
| Credentials in config | src/collectors/SECRETS.md |
Most go.d collectors are still V1, but the broad V1 authoring docs have been retired because they taught stale patterns from general Go paths. Do not use existing V1 collectors as the shape for new work.
New go.d modules MUST use V2. Start with
src/go/plugin/go.d/docs/how-to-write-a-collector.md. Use
src/go/plugin/go.d/collector/cato_networks/ as the primary modern reference,
but copy focused responsibilities rather than the entire collector. Copying a V1
module mirrors legacy patterns and the maintainers will ask you to migrate.
For migrating an existing V1 collector, start with
src/go/plugin/go.d/docs/migrate-v1-to-v2.md. Migration is compatibility work;
do not use the new-collector guide to justify chart, config, or lifecycle
contract changes. Temporary V1 parity bridges can help during development, but
the finished collector MUST NOT run through a V1-to-V2 bridge.
V2 imports: github.com/netdata/netdata/go/plugins/plugin/framework/collectorapi and .../pkg/metrix. The CollectorV2 interface lives at src/go/plugin/framework/collectorapi/collector.go.
Lifecycle semantics: Init() is one-time setup (failure disables permanently); Check() is auto-detection probe (failure disables, retried later); Collect() is the hot path (every update_every seconds); Cleanup() is guaranteed on shutdown.
Silent-failure trap (go.d). A new go.d module compiles and tests pass even when it is not loaded by the plugin at runtime. Runtime loading requires four wiring steps: import in src/go/plugin/go.d/collector/init.go, modules: toggle in src/go/plugin/go.d/config/go.d.conf, stock job config at src/go/plugin/go.d/config/go.d/<name>.conf, and entry in src/go/plugin/go.d/README.md. Same trap applies to ibm.d.
go generate after touching contexts.yaml. See src/go/plugin/ibm.d/AGENTS.md. Don't reach for ibm.d for non-IBM CGO needs — the framework is shaped around vendor drivers; CGO outside the IBM ecosystem is a design discussion.src/crates/netdata-plugin/ — modules bridge/, protocol/, rt/, charts-derive/, schema/, types/, error/. Documentation lives in lib.rs doc-comments — there is no README. New Rust crates go into the src/crates/Cargo.toml workspace. Reference impl: src/crates/netflow-plugin/.src/collectors/<name>.plugin/; reuse src/libnetdata/. libnetdata.h includes most of libnetdata so individual headers are usually unnecessary. Allocators with the z suffix (mallocz, callocz, strdupz, freez) handle failures via fatal(); freez(NULL) is safe. JSON parsing: json-c. JSON generation: buffer_json_*. Linked lists: DOUBLE_LINKED_LIST_* macros.src/plugins.d/README.md. Useful when implementation language is dictated by an SDK that go.d / ibm.d / Rust cannot accommodate.Don't:
charts.d.plugin or python.d.plugingo generate for go.d (no //go:generate directives — uses //go:embed)cd src/go && go test ./plugin/go.d/collector/<name>/...timeout 15s go run ./cmd/godplugin -m <name> -d
from src/go; success means the module registers, starts a job, and keeps
running until the timeout stops it.cargo test -p <crate>./netdata-installer.shA collector ingests one or more of these data types. Each has its own pattern.
The default. Streams as BEGIN/SET/END (PLUGINSD) or framework equivalents. Shape via NIDL (§3). Storage is the dbengine; alerts bind to chart context; anomaly detection / ML jobs run continuously. Every metric travels via streaming to parents and to Netdata Cloud — cardinality matters everywhere.
Two paths:
src/collectors/log2journal/ parses application/access logs (configurable YAML rules in log2journal.d/, e.g. nginx-json.yaml, default.yaml) and writes structured fields into the systemd journal. The systemd-journal.plugin then exposes the entries via a Function (the log explorer in the Netdata UI).src/crates/otel-plugin/ ingests OTLP logs into a write-ahead log with indexed segments (src/crates/otel-ingestor/, sfsq), queryable via the otel-logs Function in the Logs tab.Platform-specific events: windows-events.plugin (Windows event log).
Logs are not metrics. Don't try to derive metrics from logs in the collection loop — emit logs as logs, then build metrics separately if needed.
Interactive, on-demand tabular data: process lists, network connections, FDB tables, log entries, journal queries, topology snapshots, flow records. Functions complement metrics; they don't replace them.
Build a Function when the answer is interactive/tabular live data. If the answer is a numeric time series, that's a metric.
Response shape is one of info_response, data_response, topology_response, flows_response, error_response, not_modified_response (defined in src/plugins.d/FUNCTION_UI_SCHEMA.json). New topology payloads use the dedicated production topology contract in src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json. For Go, use builders in src/go/pkg/funcapi/; Go topology producers should use src/go/pkg/topology/v1 for the v1 response model and compact-table helpers. For Rust, implement the FunctionHandler trait from the SDK runtime (src/crates/netdata-plugin/rt/).
Functions run concurrently with the collection loop — they must not block it. Validate during development with src/go/tools/functions-validation/.
Reference implementations: src/collectors/network-viewer.plugin/ (topology + connections), src/collectors/systemd-journal.plugin/ (log explorer), src/collectors/apps.plugin/ (processes).
Backend docs: src/go/plugin/framework/functions/README.md (Go), src/crates/netdata-plugin/rt/src/lib.rs (Rust FunctionHandler). UI/protocol: src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md, src/plugins.d/FUNCTION_UI_REFERENCE.md. Topology contract: src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md, src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json.
Topology is its own data type — directed/undirected graphs of nodes and links. Sources and consumers:
src/go/plugin/go.d/collector/snmp_topology/) — LLDP/CDP neighbors, BRIDGE-MIB FDB, Q-BRIDGE FDB, ARP tables, STP. Builds on SNMP profiles; extending profiles is usually the right starting point.src/collectors/network-viewer.plugin/) — local L3/L4 sockets and their inferred connections.src/streaming/) — Netdata parent/child topology.src/go/pkg/topology/v1 — production Go payload
helpers for new topology producers. The non-v1 root
src/go/pkg/topology/ payload model has been retired and must not be
reintroduced for topology work.Topology is consumed via Functions (topology:* family), not via metrics. The cardinality of network edges is too high for time-series storage and the use case is interactive lookup.
When a collector needs data from another collector to enrich its output (a network collector wanting cgroup labels, an apps collector wanting cgroup PIDs, a flow collector wanting interface metadata), use netipc. Don't shell out, don't open private sockets, don't poll log files.
Both client and server roles exist in C, Go, and Rust:
src/libnetdata/netipc/src/go/pkg/netipc/src/crates/netipc/cgroups.plugin (src/collectors/cgroups.plugin/cgroup-netipc.c) is a real example of a netipc server offering cgroup metadata to other plugins. Upstream spec, tests, fuzz suite: https://github.com/netdata/plugin-ipc.
These are descriptive patterns — what existing Netdata collectors do. Use them as defaults; deviate with reason.
DB collectors often pair metrics (uptime, connections, query rates, replication lag, lock counts, cache hit ratios) with Functions for live query analysis: top queries, slow queries, currently-running queries, locks. Real examples:
src/go/plugin/go.d/collector/mysql/) — metrics + mysqlfunc/top_queries.go + processlist via collect_process_list.go.src/go/plugin/go.d/collector/postgres/) — metrics + func_top_queries.go + func_running_queries.go, dispatched through func_router.go.Before adding a query Function, decide whether it is in scope for the current work and record the product/design decision. The operator value of seeing "what's slow right now" is high and the pattern is established, but Functions are still a feature surface, not something to add accidentally during unrelated metric work.
Network/SNMP collectors typically pair metrics with topology Functions and FDB / ARP / LLDP enrichment:
snmp + snmp_topology (src/go/plugin/go.d/collector/snmp_topology/) — topology Functions (func_topology.go, func_topology_handler.go, func_topology_managed_focus.go, func_topology_options.go, func_topology_presentation.go, func_topology_depth.go) on top of SNMP profile data.network-viewer.plugin (src/collectors/network-viewer.plugin/) — topology: Functions for live socket-level topology.Per-device metrics need vnode wiring (each managed device is a vnode). FDB/ARP/STP data lands as topology Functions, not metrics — the cardinality is too high for metrics and the use case is interactive lookup.
Container collectors pair container metrics with enrichment via netipc:
cgroups.plugin exposes a netipc server (src/collectors/cgroups.plugin/cgroup-netipc.c) that other plugins query to map PIDs/cgroups to container/pod identity.apps.plugin and network-viewer.plugin consume this enrichment to label processes and connections with container metadata.When adding a new orchestration source (Kubernetes API, Docker events, Nomad, etc.), think about who downstream needs the labels and whether to expose them via netipc.
Web server collectors pair metrics (requests, status codes, latency, upstream errors) with access-log Functions when the access log is structured:
log2journal parses NGINX/Apache/HAProxy access logs (rules under src/collectors/log2journal/log2journal.d/).If the application's log format is closed or unstructured, only metrics are practical.
The Rust netflow-plugin (src/crates/netflow-plugin/) ingests flows and exposes them via Functions (flows_response shape). Flows are per-record, high-cardinality, and not suitable for traditional metric storage. Reference fixtures and provenance discipline live under src/crates/netflow-plugin/testdata/. Topology enrichment (interface names, AS metadata) typically comes from netipc or from SNMP-collected interface data.
Java app servers, message queues, application middleware — JMX/HTTP/protobuf metrics are the default; some pair with log exploration via journal or OTEL log signals when the workflow benefits from it. Mirror the closest existing collector.
Internal C plugins under src/collectors/. Reuse shared metric definitions from src/collectors/common-contexts/; follow chart-priority conventions in src/collectors/all.h; lean on src/libnetdata/ rather than reimplementing utilities.
| Topic | Open when | Path |
|---|---|---|
| NIDL framework | designing metrics, labels, charts | docs/NIDL-Framework.md |
| Chart types and dimension algorithms | choosing chart shape and metric algorithm | src/database/rrdset-type.h, src/database/rrd-algorithm.h |
| Chart priorities (C) | dashboard ordering convention | src/collectors/all.h |
| Shared metric definitions (C) | reusing common contexts | src/collectors/common-contexts/ |
| Plugin types and privileges | choosing where to add a collector | src/collectors/README.md |
| External plugin protocol | non-Go external plugin | src/plugins.d/README.md |
| go.d V2 authoring | adding a go.d module | src/go/plugin/go.d/docs/how-to-write-a-collector.md |
| go.d V1-to-V2 migration | migrating existing go.d collector | src/go/plugin/go.d/docs/migrate-v1-to-v2.md |
| Functions backend (Go / Rust) | implementing a Function | src/go/plugin/framework/functions/README.md, src/crates/netdata-plugin/rt/src/lib.rs |
| Functions UI schema & guides | response shapes and patterns | src/plugins.d/FUNCTION_UI_SCHEMA.json, src/plugins.d/FUNCTION_UI_DEVELOPER_GUIDE.md, src/plugins.d/FUNCTION_UI_REFERENCE.md |
| Topology Function schema & guide | topology actors, links, evidence, overlays | src/plugins.d/FUNCTION_TOPOLOGY_SCHEMA.json, src/plugins.d/FUNCTION_TOPOLOGY_DEVELOPER_GUIDE.md, src/plugins.d/FUNCTION_TOPOLOGY_IMPLEMENTATION_SCOPE.md |
| Functions validator | E2E + schema validation | src/go/tools/functions-validation/README.md |
| ibm.d framework | starting ibm.d work | src/go/plugin/ibm.d/AGENTS.md, src/go/plugin/ibm.d/framework/README.md |
| Rust plugin SDK | new Rust plugin | src/crates/netdata-plugin/ (rt/, protocol/, bridge/, charts-derive/, schema/, types/, error/) |
| Rust NetFlow plugin | NetFlow / sFlow / IPFIX work | src/crates/netflow-plugin/ |
| OTEL ingestion mappings | per-metric YAML routing | src/crates/otel-ingestor/ (configs under configs/otel.d/v1/metrics/) |
| SNMP profile format | adding/extending an SNMP profile | src/go/plugin/go.d/collector/snmp/profile-format.md |
| SNMP stock profiles | starting from a known device | src/go/plugin/go.d/config/go.d/snmp.profiles/default/ |
| statsd synthetic_charts | operator-curated dashboards | src/collectors/statsd.plugin/README.md (lines 397-639) |
| Prometheus mapping | generic exposition scrape | src/go/plugin/go.d/collector/prometheus/README.md |
| Prometheus profile format | curated exporter dashboards + autogen fallback selectors | src/go/plugin/go.d/collector/prometheus/profile-format.md |
| Prometheus metric relabeling | rewriting scraped metric names/labels | src/go/plugin/go.d/collector/prometheus/relabel/README.md |
| log2journal | parsing application logs into the journal | src/collectors/log2journal/log2journal.d/ |
| Auto-discovery rules | adding service-detection rules | src/go/plugin/go.d/config/go.d/sd/{net_listeners,docker,snmp,http}.conf |
| Topology library | topology producers in Go | src/go/pkg/topology/v1 |
| netipc cross-plugin enrichment | C / Go / Rust | src/libnetdata/netipc/, src/go/pkg/netipc/, src/crates/netipc/ |
| DYNCFG protocol | dynamic configuration | src/plugins.d/DYNCFG.md, docs/developer-and-contributor-corner/dyncfg.md |
| Health alerts reference | alert template authoring | src/health/REFERENCE.md, src/health/alert-configuration-ordering.md |
| Integrations pipeline | doc generation from metadata.yaml | integrations/README.md |
| Go framework changes | changing shared Go collector/runtime framework code | src/go/plugin/framework/docs/changing-framework-code.md |
| go.d V1-to-V2 migration | migrating existing go.d collectors | src/go/plugin/go.d/docs/migrate-v1-to-v2.md |
| Credentials in config | ${env:}/${file:}/${cmd:}/${store:} | src/collectors/SECRETS.md |
| Privileged operations | restricted setuid helper | src/collectors/utils/ndsudo.c |
This skill is live. When you find a gap, an outdated pointer, a new pattern, or a bad practice not yet captured, propose changes to this file in the same PR that exposed the issue. When fixing a wrong pointer, also record what was misleading about the prior text — future readers see both the corrected map and the failure mode that produced it. Mention the change in the PR description so it gets reviewed consciously rather than skimmed.