packaging/installer/SUPPORT-BUNDLE.md
netdata-support-bundle collects a sanitized diagnostic bundle (tarball on POSIX
systems, zip on Windows) that users attach to support tickets, so support gets
everything it needs on first contact instead of asking for it over multiple
round trips.
netdata-support-bundlenetdata-support-bundle.ps1The POSIX file is intentionally extension-less (netdata-support-bundle, not
.sh): the installed command and its repository path stay implementation-neutral,
so a future reimplementation (e.g. in Rust) can replace it at the same path and
command name without changing any build/packaging reference. It is a /bin/sh
script today (see the shebang). The Windows counterpart keeps .ps1 because
PowerShell requires the extension to execute a script; on Windows a future
replacement is shipped through the MSI packaging instead.
# installed with the agent (in PATH, like netdatacli):
sudo netdata-support-bundle
# static installs:
sudo /opt/netdata/usr/sbin/netdata-support-bundle
# For an older agent, download from an immutable release tag that contains the
# tool. Never pipe a URL into a root shell and never use the mutable master ref.
tag='<TRUSTED_NETDATA_RELEASE_TAG>'
t=$(mktemp "${TMPDIR:-/tmp}/netdata-support-bundle.XXXXXX")
trap 'rm -f "$t"' EXIT HUP INT TERM
curl -fsSL -o "$t" "https://raw.githubusercontent.com/netdata/netdata/$tag/packaging/installer/netdata-support-bundle" \
|| wget -qO "$t" "https://raw.githubusercontent.com/netdata/netdata/$tag/packaging/installer/netdata-support-bundle"
Inspect the downloaded script. Only after that separate review, run:
sudo sh "$t"
# Windows (elevated PowerShell):
powershell -ExecutionPolicy Bypass -File "C:\Program Files\Netdata\usr\libexec\netdata\netdata-support-bundle.ps1"
Both scripts implement the same bundle contract: same directory layout, same
MANIFEST.json schema (netdata-support-bundle/v1), same sanitization rules.
If you change one script, mirror the change in the other and update this
document.
| guarantee | implementation |
|---|---|
| Zero system impact | self-demotion to idle CPU/IO priority (nice -n 19 + ionice -c 3 / PriorityClass = Idle); per-command timeout (10 s default, via timeout or a portable watchdog — the watchdog kills the direct child only, a documented limitation); global deadline checked before each collector, so the hard runtime bound is deadline + one command timeout; size caps (5 MiB per log, 1 MiB per file, 2 MiB per command/API output); read-only — writes only its private staging dir and the final artifacts, never restarts or reconfigures anything; artifacts are published with O_EXCL so pre-existing files or symlinks in shared tmp dirs are never followed |
| Works when the agent is dead | no hard dependency on a running agent; the most valuable crash artifacts (status file, logs, buildinfo via the binary) are collected from disk; a 07-runtime/AGENT-WAS-DOWN.txt marker is written instead of API captures |
| Secrets always redacted | non-optional single-pass sanitizer; see "Sanitization" below |
| PII pseudonymized by default | IPs (v4+v6), MACs, emails, this host's names, the invoking user, child/mirrored node hostnames and stream destinations are replaced with stable pseudonyms (ip-1, private-host-1) so cross-file correlation still works; the private map is saved next to the bundle, never inside it; --no-obfuscate / -NoObfuscate opts out |
| Caps cannot expose secrets | all caps cut at LINE boundaries, so a secret can never straddle the cut and dodge the line-based sanitizer; a capped tail with no line break at all is withheld entirely; sanitizer failures withhold the file content (fail closed) |
| Legible to humans AND AI agents | triage-ordered numbered directories; sanitized file copies have no injected provenance headers; provenance headers only on command captures; MANIFEST.json indexes every file with safe origin + sanitization state; summary.txt opens with a triage read-order |
| platform | status | notes |
|---|---|---|
| Linux (glibc, systemd) | tested | full collection incl. journal namespace |
| Linux (musl/BusyBox, e.g. Alpine) | tested | BusyBox timeout has no -k (auto-detected); file logs instead of journal |
| Docker (official image) | tested | agent logs live in docker logs on the host — bundle includes a marker with the exact command to run and attach; /proc/1/environ needs CAP_SYS_PTRACE (fallback to exec env) |
Static builds (/opt/netdata) | tested paths | all paths resolved under the prefix |
| FreeBSD | best effort | /usr/local/etc/netdata + /var/db/netdata paths, sockstat fallback, ps -H threads; no /proc items |
| macOS (Homebrew) | best effort | /usr/local and /opt/homebrew prefixes, sysctl/vm_stat fallbacks, ps -M threads |
| Windows | tested in CI | Windows PowerShell 5.1 runs the adversarial suite and builds/opens a complete fixture zip; PowerShell 7 runs the same sanitizer suite |
Portability rules the POSIX script obeys (keep them when editing):
sh only — no bashisms (users run it with sh, dash, BusyBox ash).{n,m} regex intervals inside the awk sanitizer — older BSD awks
treat them as literal braces, which would silently disable redaction.
Character classes are written out explicitly instead.ionice, journalctl, coredumpctl,
curl/wget, ss/sockstat/netstat,
free/sysctl, /proc availability are all probed before use.Analysis of the full Freshdesk ticket history (443 tickets, 2026-07) and 287 maintainer comments across 79 GitHub bug threads showed:
Every item collected maps to a recurring support ask. That mapping is the "why" column in the tables below. When adding a new item, add its why.
summary.txt, MANIFEST.json, README.md (bundle root)| item | why |
|---|---|
summary.txt | one-page human overview; opens with agent state and a "read order for triage" per issue class, so support (or an AI agent) starts at the right file |
MANIFEST.json | machine-readable index: every file with its origin (command / source path / API endpoint), size, and sanitization state; lets AI tooling navigate the bundle without guessing |
README.md | self-documentation for whoever receives the bundle |
01-system/ — platform context| item | why |
|---|---|
| kernel/OS/architecture, distro | first question in the bug template; kernel regressions have been root causes (two GitHub issues traced to kernel changes) |
| memory, disks, CPU count, uptime | capacity questions asked in most performance tickets |
| virtualization / container detection, cgroup version | OpenVZ/LXC/CageFS visibility problems are a recurring collector-failure class |
| clock/time sync | clock drift on children silently breaks streaming and cloud auth — maintainers explicitly ask ("check if the clock on child nodes is drifting") |
/proc/self/mountinfo (POSIX) | namespace visibility issues ("cannot open /proc/diskstats") are diagnosed from the mount table |
| kernel OOM/segfault messages | evidence of the kernel killing netdata — distinguishes crashes from kills |
| SELinux/AppArmor state | MAC denials cause silent collector failures |
02-install/ — how netdata got here| item | why |
|---|---|
.environment file | install method, flags, release channel, custom CFLAGS (-ffast-math alone broke dbengine once); contains no secrets |
.install-type marker | kickstart-build / kickstart-static / oci / binpkg-* — determines which update/troubleshoot paths apply |
| package manager info | version skew between repo package and expectation is a recurring theme |
| container context (env, cgroup, pid 1) | missing init: true, missing pid: host, and wrong images are recurring Docker-ticket root causes; NETDATA_* env values pass through the sanitizer |
03-process/ — the running agent| item | why |
|---|---|
| netdata process tree with CPU/memory | "netdata is eating my CPU/RAM" tickets need this first |
| per-thread CPU (POSIX) | maintainers ask users to find the hot thread in htop; this captures it non-interactively |
/proc/PID/status, limits, fd count | leak and limit diagnosis |
| agent process environment (sanitized) | proxy/claiming issues: the env the service sees differs from the user's shell — asked explicitly in GitHub threads |
| zombie process check | plugin-reaping failures in containers (init: true guidance) |
04-config/ — configuration| item | why |
|---|---|
effective running config (GET /netdata.conf) | the #1 GitHub maintainer ask; shows the merged config the agent actually uses and annotates unrecognized options — resolves "my config is ignored" outright; authoritative over on-disk files |
on-disk netdata.conf, stream.conf, cloud/claim conf, go.d.conf, go.d/health.d/python.d/charts.d/statsd.d user files, exporting.conf | the files users were asked to paste, ticket after ticket (child stream.conf + parent [web]/[stream] sections is a canned Freshdesk ask); all pass the sanitizer, and their bundle paths mirror their paths relative to the config directory |
05-logs/ — history| item | why |
|---|---|
systemd journal, including --namespace=netdata | the agent logs to its own journal namespace on systemd installs — plain journalctl -u netdata misses almost everything; support asks for "a complete log from start until the problem" |
/var/log/netdata/*.log tails (size-capped) | non-systemd installs, static builds, macOS/BSD |
Windows Event Log (NetdataWEL + Application, Netdata providers) | Windows agents log to the Event Log; Windows is a top-3 support theme |
| updater service journal | update failures; the updater keeps no persistent log file |
coredump metadata (coredumpctl list, never the dumps) | tells support a dump exists and matches the crash time — the dump itself is fetched later only if needed |
| docker marker file | in containers the log "files" are symlinks to stdout — history only exists in docker logs on the host; the bundle says raw logs must not be attached and gives a private capture/review/redaction workflow using the requested time window |
06-state/ — persistent state| item | why |
|---|---|
status-netdata.json (trusted fallback locations, newest safe file wins; shared /tmp is excluded) | the single most valuable crash artifact: last exit reason, fatal line/file/function, signal, stack trace — same data that feeds agent-events crash telemetry; support gets crash forensics with zero extra round trips |
| state dir aggregate inventory | unexpected file counts/sizes without exposing filenames that may themselves be live tokens, hostnames, or job identifiers; contents of secret files are never read (see exclusions) |
claim state (claimed_id only) | claim id is the identifier support needs to find the node in Cloud; a non-persisted cloud.d across restarts is a known Freshdesk root cause |
| db disk usage per tier + sqlite sizes | retention questions ("why do I only have N days") are answered by tier sizes vs configured limits |
| dyncfg files (sanitized) | jobs created via UI live here, not in /etc/netdata — invisible in classic config collection |
| go.d job statuses, health silencers | which collector jobs exist/fail; why alerts are silent |
07-runtime/ — live agent state (only when API responds)| item | why |
|---|---|
/api/v3/info | best single call: structured buildinfo, features, cloud status, per-tier retention — and it works even under bearer protection |
/api/v1/info, /api/v2/node_instances | children, streaming state, db_size, metric counts — the exact endpoint maintainers ask for in retention/memory tickets |
/api/v3/stream_info, /api/v1/aclk | streaming and cloud-connection diagnostics |
| active alerts + alert instances | alert tickets are the single biggest Freshdesk theme |
/api/v1/functions, /api/v1/ml_info | which plugins expose what; ML state |
netdata -W buildinfo + buildinfojson | required by the bug template; the paths section proves which config dirs the binary uses; works with the daemon down |
netdata -W cmakecache | authoritative record of how the agent was built (compiler flags, enabled/disabled plugins, configured paths) — a superset of buildinfo; pinpoints build-time causes (a disabled plugin, a custom flag) that buildinfo alone can miss |
netdata -W perflibdump -perflibfile (Windows) | performance-library counter/instance metadata — the most common Windows support class is perflib-related (e.g. PerflibSMB); collected as a file so a large dump is not truncated, and sanitized like any other file. Reads HKEY_PERFORMANCE_DATA, so it needs an elevated session and can be slower than other commands: the collector pre-checks elevation, uses a 30s timeout floor, and never drops it silently — when it cannot run, perflib.json holds a JSON marker stating why (not elevated, timed out, or an access error) so support sees the reason instead of a missing file |
netdatacli aclk-state json | canned Freshdesk ask for cloud issues |
| netdata self CPU/memory/clients CSVs (10 min, bounded) | replaces the "please send a screenshot of the Netdata memory charts" round trip |
Local API reads target 127.0.0.1:19999 directly and bypass any configured
proxy, so diagnostic data cannot leave the host through a forced proxy.
08-network/ — connectivity| item | why |
|---|---|
| listening sockets (netdata-related) | "dashboard unreachable" and port-conflict tickets |
| DNS config, proxy env/config (sanitized) | claiming-behind-proxy is a recurring theme; DNS misconfiguration breaks cloud connectivity |
| Netdata Cloud reachability (TCP plus certificate-validating HTTPS/TLS probe; no bundle data sent) | separates network problems from agent problems in one step |
These are excluded by design. Do not add them.
cloud.d/private.pem (ACLK private key), cloud.d/token (claim token)bearer_tokens/ (the filenames are live API tokens), netdata.api.key,
mcp_dev_preview_api_key, netdata_random_session_id/etc/netdata/ssl/ and any *.pem / *.keyml.db, registry.db (person GUIDs
and dashboard URLs)This tool follows the same proportionate posture as established support-bundle
tools (sosreport, supportconfig, kubectl cluster-info dump, Elastic's
diagnostics): redact the well-defined, high-value cases robustly, and treat
redaction as best-effort defense-in-depth — not a guarantee.
Two facts do the heavy lifting and are why we do not chase completeness:
summary.txt and this document say so).Concretely, we redact credential-bearing config keys, URL/DSN credentials,
JWT/Bearer/Basic tokens, PEM key blocks, stream.conf API-key sections, and
PII (IPs, MACs, emails, hostnames, usernames); and we never collect files that
are pure secrets at all (the never-collect list). We deliberately do not
try to parse arbitrary nested structure to prove no secret can ever slip
through — a line-based tool cannot balance nested JSON brackets or detect
indentation-based YAML block-scalar boundaries reliably, and every attempt adds
fragile regex for encodings that do not occur in the data this bundle collects.
A brittle sanitizer that tries to do everything is worse than a stable one that
does the common cases well; the durable place for structure-aware,
schema-driven redaction is inside the agent, not a portable shell/PowerShell
script. When extending the tool, prefer this restraint.
Two passes, one sweep, applied to every collected file:
api key, apikey, token, password, passwd, pwd, secret, community, bearer, webhook, license key, auth, credential, cookie, passphrase, proxy user, proxy pass, username, dsn, private key, access key, session, recipient, account sid, priv key — in ini (k = v), yaml (k: v), env (K=V) and
JSON ("k": "v") forms, covering escaped JSON strings and numeric/scalar
JSON values. (Aliases are matched as substrings, so only unambiguous
secret tokens are on the list — e.g. pat is deliberately NOT, because it
matches path.) Keys must look like real config keys (≤64 chars,
no sentence punctuation) so prose containing "token" is not mangled.
Exemptions are decided by the KEY, never the value: keys ending in
file path dir directory protection support mode level port timeout cookies secure log size options format type describe secrets rather than being
secrets, so bearer token protection = no and api key file = /path
stay readable while TOKEN=false and PASSWORD=/x are redacted;-token=X, --password "X",
CLAIM_TOKEN=X, api key = X inside captured process command lines),
including single- and double-quoted values;scheme://user:pass@) and Go DSN credentials
(user:pass@tcp(...));Bearer <value> where the value contains a digit (real tokens do;
this avoids mangling config prose like bearer token protection = no),
Basic <value>, and Authorization: header values;?token=, &api_key=, ... — request
lines in access logs);stream.conf parent-side [<UUID>] section headers (they ARE the API
keys);bearer_tokens/ directory listings show a file COUNT only — the
filenames are the tokens.--no-obfuscate / -NoObfuscate to disable:
ip-N and IPv6 → ip6-N (stable per
bundle; compressed, lettered, and numeric-only uncompressed forms;
validated so timestamps, file.c:123 refs and ::1 are left alone);[MAC]; email addresses → [EMAIL];redacted-host; the invoking user's name →
redacted-user;private-host-N; only public Netdata service domains and
a small exact allowlist of known Netdata filenames are preserved (a broad
suffix exemption would leak names such as customer.key);stream.conf destination hosts regardless of TLD → private-host-N;search/domain values → [SEARCH-DOMAINS-WITHHELD]
(corporate search domains are rarely under private TLDs).The private map is written next to the bundle (*.pseudonym-map.tsv) so the
user can decode references if support asks "what is private-host-2?" — it
is never included in the bundle itself. Pseudonym mappings are capped at 4096
entries; past the cap, values get a non-correlating placeholder so hostile
high-cardinality input cannot grow memory or the private map without bound.
Redaction here is defense in depth, not a substitute for exclusion: files that
are pure secrets (see exclusion list) are never read at all. Files containing
NUL bytes (binary or BOM-less UTF-16 input) are withheld rather than run
through byte-unsafe line redaction. A source file whose leaf is itself a
symlink is withheld (a swapped link must not redirect collection to another
target); symlinked parent directories resolve normally. Do not extend
collection to a
directory writable by any other identity. Each run uses a newly created,
unpredictable staging directory and never reuses a pre-existing path. POSIX
staging is created under umask 077 on POSIX and under the per-user %TEMP%
tree on Windows, with an unpredictable random name. Final artifacts (tarball,
zip, pseudonym map) are published with a no-overwrite move so a pre-existing
file or symlink at the target is never followed or clobbered.
The agent itself provides no redaction anywhere — GET /netdata.conf and
netdatacli dumpconfig print secrets verbatim. Everything must be sanitized
by these scripts.
collect_cmd / collect_file / collect_api
(Save-Cmd / Save-File / Save-Api / Save-CmdRaw on Windows). They enforce
timeouts, size caps, sanitization, and manifest registration. Never write
into the bundle directly..sh ↔ .ps1) or record explicitly
in your PR why it is platform-specific.netdata-support-bundle --selftest (netdata-support-bundle.ps1 -SelfTest on Windows) — it must
pass on GNU awk, mawk, BusyBox awk, and PowerShell. CI executes both suites.
For new collection sources also
plant a sentinel secret in the source, run a collection, and grep -r the
extracted bundle. Zero hits or it does not ship.netdata-support-bundle/v1 (in MANIFEST.json). Bump the suffix on
breaking layout changes; downstream ticket tooling may parse it..txt files starting with a
# netdata-support-bundle v<version> | command: ... | captured: <utc> header; on POSIX
they also end with an # exit: N | duration: Ns trailer. PowerShell command
captures carry the provenance header only (background jobs do not surface a
meaningful process exit code).MANIFEST.json, not in the files.