Go Helper Packages For go.d Collectors

Use existing helper packages before adding collector-local plumbing. A helper is not better because it is shared; it is better when it gives users the same configuration shape, the same safety behavior, or the same testable parsing path as other collectors.

This guide covers helper surfaces used by go.d collectors across:

• src/go/pkg/* for shared Go packages used beyond go.d;
• src/go/plugin/go.d/pkg/* for go.d-specific helpers;
• src/go/logger for the logger embedded through collectorapi.Base.

It is not an exhaustive API reference. Before adding a local helper, search these roots for an existing package that already owns the behavior.

Helper Roots

Config Option Types

Use src/go/pkg/confopt for common configuration value types.

When:

• users configure durations that should accept strings such as 5s, 30m, or numeric seconds;
• users need explicit auto / enabled / disabled behavior instead of a plain boolean;
• a migration needs to preserve legacy pointer-boolean semantics without keeping pointer plumbing in new code.

Why:

• confopt.Duration and confopt.LongDuration centralize YAML/JSON duration parsing and formatting;
• confopt.AutoBool makes tri-state behavior explicit and schema-friendly;
• collectors avoid ad hoc parsers and inconsistent boolean defaults.

HTTP Collectors

Use src/go/pkg/web for HTTP-based collectors.

When:

• the collector talks to an HTTP or HTTPS endpoint;
• users need the normal Netdata HTTP options: url, timeout, redirects, proxy, basic auth, bearer token file, headers, body, method, and TLS fields;
• the collector builds repeated requests against the same endpoint.

Why:

• web.HTTPConfig embeds web.RequestConfig and web.ClientConfig so HTTP collectors expose the same option surface;
• web.NewHTTPClient(c.ClientConfig) applies timeout, TLS, proxy, redirect, and HTTP/2 behavior consistently;
• web.NewHTTPRequest(c.RequestConfig) and web.NewHTTPRequestWithPath(c.RequestConfig, path) apply user agent, authentication, headers, body, and safe path joining.

Pattern:

type Config struct {
    web.HTTPConfig `yaml:",inline" json:""`
}

Use src/go/pkg/tlscfg directly only when the collector is not HTTP-based but still needs TLS, such as Redis or x509-style checks. HTTP collectors should get TLS behavior through web.HTTPConfig.

Configured credential and TLS files

web bearer-token files and tlscfg CA files use src/go/pkg/safefile; certificate and key files use it when both are configured. The helper opens the path once, verifies the opened object is a regular file, reads at most 1 MiB, and closes it. Symlinks to regular files are supported; non-regular objects and larger files are rejected.

Use safefile.Read for new bounded credential or key-material paths that share this contract. Do not add a separate preflight followed by os.ReadFile: that checks a different filesystem object and leaves the production read unbounded.

Prometheus Endpoints

Use src/go/pkg/prometheus when the upstream endpoint exposes Prometheus text format.

When:

• the collector scrapes /metrics or another Prometheus exposition endpoint;
• the collector needs to parse metric families or sorted series;
• the collector needs a bounded selector for metric names.

Why:

• it reuses web.RequestConfig and *http.Client;
• it handles Prometheus text parsing and gzip responses;
• selectors avoid parsing or processing metric families the collector will not use.

Do not hand-roll text exposition parsing in a collector.

Selectors And Matchers

Use src/go/pkg/matcher for user-facing include/exclude or selector fields.

When:

• users select entities by name, ID, interface, queue, topic, or similar labels;
• the selector syntax can be glob, regexp, string, or simple patterns;
• negative matches such as !*test* * are sufficient.

Why:

• users get one matcher grammar across collectors;
• tests can cover selector behavior without custom parser logic;
• existing logical matchers can combine conditions when needed.

Do not invent a selector language unless the upstream API requires one. Prefer a single simple-pattern field for simple cases; add separate include/exclude fields only when the user problem needs that shape.

Do not use src/go/pkg/selectorcore for user-facing collector selectors. It is the lower-level selector metadata/parser surface used by template and selector engines, not the normal collector selector helper.

Limited Logging

Collectors embed collectorapi.Base, which embeds *logger.Logger. Use the logger's built-in limiting before adding collector-local rate-limit state.

When:

• an error can repeat every collection cycle;
• a partial failure is useful to report but would spam logs;
• a one-time notice or warning is enough.

Why:

• in go.d jobs, c.Once(key).Warningf(...) is cycle-local because the runtime resets Once state each runOnce; it is useful for suppressing duplicate messages inside one cycle only;
• c.Limit(key, n, window).Warningf(...) logs at most n messages per key per window and is the right default for cross-cycle spam control;
• the limiter is shared through the collector logger and already used by modern collectors such as Cato Networks, PAN-OS, and vSphere.

Pattern:

c.Limit("mycollector:operation:error", 1, time.Hour).
    Warningf("operation failed: %v", err)

Use stable keys. Include the operation and bounded error class when needed, but do not put unbounded IDs, URLs, query strings, customer names, or raw provider messages in the key.

Custom warning gates are justified only when the built-in count-per-window semantics are not the right behavior, for example when logging only on state transitions. Document that reason in the PR description or design note so reviewers can see why the built-in limiter was not enough.

Socket Clients

Use src/go/plugin/go.d/pkg/socket for simple TCP, UDP, or Unix-socket line-protocol collectors.

When:

• the collector connects to a local or remote socket and sends text commands;
• the response is processed line by line;
• the collector needs shared timeout, TLS, and max-read-line behavior.

Why:

• socket address parsing is shared across collectors;
• connect, command, read, disconnect, deadline, and line-limit behavior stay consistent;
• tests can use the helper's fake TCP/UDP/Unix servers instead of custom socket harnesses.

Do not hand-roll socket dial/read loops for common line-oriented protocols.

External Commands

Use src/go/plugin/go.d/pkg/ndexec for collectors that execute binaries.

When:

• the collector needs a local command output;
• the command should run through Netdata's helper wrappers;
• the command may need privilege through ndsudo;
• tests need to stub helper paths.

Why:

• arguments are passed without a shell;
• timeouts and context cancellation are handled;
• stderr snippets are bounded;
• helpers integrate with Netdata's execution model.

Use:

• RunUnprivileged / RunUnprivilegedWithOptions... for unprivileged commands;
• RunNDSudo for commands exposed through ndsudo;
• RunDirect only when direct execution is intentionally required;
• FindBinary for PATH/default-path discovery.

Do not call exec.Command directly unless the helper cannot support the case and the reason is documented.

Log File Collectors

Use src/go/plugin/go.d/pkg/logs for collectors that parse application log files.

When:

• the collector tails files that can rotate;
• the log format is CSV, LTSV, regexp, or JSON;
• parser errors should be distinguishable from I/O errors.

Why:

• logs.Reader is log-rotation aware;
• logs.NewParser centralizes supported parser types;
• logs.IsParseError lets collection logic treat malformed rows differently from source failures.

Do not open and seek log files manually unless the collector's source is not a normal file-tail workflow.

IP Ranges

Use src/go/plugin/go.d/pkg/iprange when users configure address ranges.

When:

• the collector filters IPs, networks, peers, or hosts by ranges;
• the config accepts CIDR, range, or other supported range syntax.

Why:

• range parsing and membership checks are shared;
• invalid syntax handling is consistent;
• collectors avoid slightly different IP matching semantics.

SQL Helpers

Use src/go/plugin/go.d/pkg/sqlquery for repeated SQL row-scanning patterns.

When:

• the collector or Function scans rows into strings, integers, floats, or discard columns;
• the collector needs table-column discovery with ? or $1 placeholders;
• the row-to-value assignment is generic across queries.

Why:

• scan holders and null handling are centralized;
• query duration measurement and row iteration behavior stay testable;
• Function code can avoid custom one-off scanners.

Cloud Auth Helpers

Use src/go/plugin/go.d/pkg/cloudauth when a cloud collector needs supported cloud-provider credentials.

When:

• the collector supports cloud_auth configuration;
• Azure AD credential construction is needed.

Why:

• provider names normalize consistently;
• validation is centralized;
• unsupported providers fail with consistent errors.

Ping Helpers

Use src/go/plugin/go.d/pkg/pinger for ping/latency probing.

When:

• a collector needs ICMP-style probing;
• it needs shared latency/jitter derivation.

Why:

• probe config validation and derived metrics are shared;
• collectors avoid reimplementing packet sampling and jitter math.

Profile Catalog Helpers

Use src/go/plugin/go.d/pkg/profilecatalog when a collector ships curated per-target profile files and loads them from stock plus user directories. By default a profile's identity is its YAML filename without the extension; collectors with compound encodings can supply their own filename-to-identity parser. Used by the prometheus, azure_monitor, cloudwatch, and snmp_traps collectors.

When:

• the collector reads profiles from config/go.d/<name>.profiles/ (stock) and the user config dirs;
• it needs stock/user override precedence (user overrides stock by logical identity), stock-fatal errors, and either skip-invalid-user or fail-invalid-user behavior;
• it may need the optional process-wide cache, or may own a shorter catalog lifecycle itself.

Why:

• one shared Load[P] + Catalog[P] + Cached[T] replaces per-collector copies of the directory walk, override precedence, and singleton caching;
• it is generic over the collector's profile type P and oblivious to matching (matching stays in the collector);
• loading depth is the collector's choice: Options.Decode receives file bytes, while Options.LoadFile lets the caller own compression, size limits, or path-based lazy state;
• Options.ParseFileName can derive one logical identity from compound suffixes while preserving the default YAML behavior for existing callers.

Do NOT put matching logic in this package; it is a catalog + loader, not a matcher. Keep the profile schema, its decode/validate, the defaultDirSpecs directory resolution (location-specific), and specialized queries in the collector's own profile package. A collector may wrap profilecatalog.Catalog[P] when it needs specialized queries.

Reverse DNS

Use src/go/plugin/go.d/pkg/reversedns when multiple collectors or jobs need PTR data from one bounded process-owned cache.

Choose the API by caller behavior:

• Lookup is cache-only and performs no DNS I/O.
• Schedule is best-effort and non-blocking; use it from per-item hot paths.
• Resolve waits for a cached or coalesced lookup; use it from background warmers and other blocking paths.

The resolver canonicalizes mapped IPv4 addresses, normalizes PTR names deterministically, caches positive and negative results with separate TTLs, coalesces work by address, and bounds both active lookups and retained entries. Blocking Resolve work receives admission priority over new Schedule work. Its segmented retention policy protects repeatedly used positive entries from one-pass source scans.

Create the resolver at the composition root and inject the same pointer into its consumers. Collectors borrow it: they MUST NOT close, sweep, or replace it during per-job lifecycle. Keep collector-specific address eligibility, candidate selection, display precedence, and audit mapping in collector-owned adapters rather than adding those policies to the generic package.

Legacy V1 Helpers

src/go/pkg/stm converts structs into map[string]int64. src/go/plugin/go.d/pkg/oldmetrix provides V1 metric vector helper types such as counters, summaries, histograms, and boolean conversions used by existing V1 collectors. Both helpers are V1-shaped. New V2 collectors MUST NOT use them as their metric path.

Acceptable uses:

• maintaining an existing V1 collector;
• temporary parity tests during V1-to-V2 migration, provided the helper is not reachable from the final runtime path.