Query Netdata Agents directly

This skill teaches end-users (and AI assistants helping them) how to talk to a Netdata Agent's HTTP API directly, including bearer-protected agents that require an SSO-issued bearer token.

It is the sibling of query-netdata-cloud. The two skills cover different transports for the same underlying agent API.

Index of guides

For what to query (function payloads, body schemas), see the sibling skill -- the agent and the Cloud proxy expose the same Function payload shape.

This skill ships shell scripts at scripts/_lib.sh that automate the bearer mint / cache / refresh / call-function flow. End-users can either use the scripts as a black box, or read the script source as a reference implementation.

Mandatory Requirements (READ FIRST)

1. If you analyze, you author a how-to. When asked a concrete question about an agent that isn't already covered by an existing how-to under how-tos/, you MUST author a new how-to and add it to how-tos/INDEX.md BEFORE completing the task. The catalog is live -- the next assistant should not redo the same analysis. Keep this catalog operator-facing: recipes here should explain how to fetch or use Agent data. Developer contract validation for topology producers, schemas, fixtures, UI adapters, or aggregator handoffs belongs in the relevant project developer skill, not in this public skill.
2. Use the token-safe wrappers. agents_query_cloud, agents_query_agent, agents_call_function from scripts/_lib.sh handle auth internally and emit only the response body to stdout. Never write raw curl with a literal Authorization: Bearer $TOKEN or X-Netdata-Auth: Bearer <uuid>. Bearers / cloud tokens / claim_ids must NEVER reach assistant-captured stdout.
3. Provide actionable instructions. End every recommendation with a runnable wrapper invocation.
4. Never request credentials. Use env-key placeholders (NETDATA_CLOUD_TOKEN, AGENT_EVENTS_HOSTNAME, AGENT_EVENTS_NODE_ID, etc.) -- the user fills .env locally.
5. Bearer values stay in .env and .local/. The bearer cache file at <repo>/.local/audits/query-netdata-agents/ bearers/<machine_guid>.json is mode 0600 and gitignored. The internal helper _agents_resolve_bearer returns it through a validated caller-local output variable, never to stdout.
6. For bearer-protected agents, default to the Cloud-token flow in this skill (it auto-mints + caches the bearer).

Prerequisites

• All SKILL.md prereqs from query-netdata-cloud: cloud token, space ID, room ID, node UUID.
• Network access to the agent on port 19999 (or whatever it binds). Test with: curl -sS http://AGENT_HOST:19999/api/v3/info -- a 200 with JSON confirms reachability.
• The agent's claim_id if you intend to mint a bearer. It's at /api/v3/info -> .agents[0].cloud.claim_id, or with shell access at <netdata-prefix>/var/lib/netdata/cloud.d/claimed_id. For the install-prefix detection rule, see scripts/_lib.sh.

.env keys consumed (none are added by this skill -- the four existing AGENT_EVENTS_* keys cover the maintainer-facing agent-events workflow):

Detect bearer protection

The signal is HTTP 412 Precondition Failed from the agent for any authenticated path (e.g. /host/<uuid>/api/v3/function?...). The response body is You need to be authorized to access this resource.

# Probe -- 412 means bearer required, 200 means open access
HOST="agent.example.invalid:19999"
NODE="YOUR_NODE_UUID"

curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  -H 'Content-Type: application/json' \
  "http://$HOST/host/$NODE/api/v3/function?function=systemd-journal" \
  -d '{"info":true}'

The unauthenticated /api/v3/info endpoint is always reachable (returns 200 with the agent's identity). Use it to confirm the host is up before checking auth.

Mint a per-agent bearer

Endpoint: GET /api/v2/bearer_get_token on Netdata Cloud.

Required query parameters: node_id, machine_guid, claim_id. Auth: Cloud token in Authorization: Bearer ....

TOKEN="YOUR_API_TOKEN"
NODE_ID="YOUR_NODE_UUID"
MACHINE_GUID="YOUR_MACHINE_GUID"
CLAIM_ID="YOUR_CLAIM_ID"

curl -sS \
  -H "Authorization: Bearer $TOKEN" \
  "https://app.netdata.cloud/api/v2/bearer_get_token?node_id=$NODE_ID&machine_guid=$MACHINE_GUID&claim_id=$CLAIM_ID"

Response body:

Permission gate (Cloud-side): PermissionSpaceRead on the target space; node must be reachable. If the agent is stale, the call returns 400.

Use the bearer to call an agent

HOST="agent.example.invalid:19999"   # the agent's bind address
NODE="YOUR_NODE_UUID"                # the node UUID (== nd field)
BEARER="MINTED_BEARER_UUID"

curl -sS -X POST \
  -H "X-Netdata-Auth: Bearer $BEARER" \
  -H 'Content-Type: application/json' \
  "http://$HOST/host/$NODE/api/v3/function?function=systemd-journal" \
  -d '{"info":true,"timeout":30000}'

Notes:

• The header is X-Netdata-Auth: Bearer ..., NOT Authorization: Bearer .... The agent rejects the latter for per-agent bearer auth.
• The agent's HTTP API path mirrors the Cloud-proxied path. For the Function payload shape (e.g. systemd-journal query body), see the matching guide in query-netdata-cloud.

Bearer cache and refresh

The shipped scripts cache bearers per machine_guid under <repo>/.local/audits/query-netdata-agents/bearers/<machine_guid>.json (gitignored, mode 0600). Each cache entry stores the raw mint response.

Refresh policy: the cache is considered expired when expiration - now < 3600 (one-hour buffer before actual TTL). Mirror of the Cloud frontend's policy (cloud-frontend/src/domains/nodes/useAgentBearer.js).

A failed mint clears the cache entry so the next call re-mints from scratch.

Scripts

The reference implementation lives in scripts/_lib.sh. It exposes token-safe public wrappers (the assistant never sees the cloud token, agent bearer, or claim_id on stdout) and a self-test that asserts no token bytes leak.

# In your script:
source "$(git rev-parse --show-toplevel)/.agents/skills/query-netdata-agents/scripts/_lib.sh"
agents_load_env

# Cloud-side call. NETDATA_CLOUD_TOKEN is read from .env
# internally; stdout is the response body only.
agents_query_cloud GET /api/v2/spaces

# Direct-agent call. The bearer is minted/cached/refreshed
# internally. stdout is the response body only; stderr shows the
# curl invocation with `<CLOUD_TOKEN>` and `<AGENT_BEARER>`
# masked.
agents_query_agent \
    --node "$AGENT_EVENTS_NODE_ID" \
    --host "$AGENT_EVENTS_HOSTNAME:19999" \
    --machine-guid "$AGENT_EVENTS_MACHINE_GUID" \
    POST '/api/v3/function?function=systemd-journal' '{"info":true}'

# Convenience: pick transport with --via cloud|agent.
agents_call_function \
    --via cloud \
    --node "$AGENT_EVENTS_NODE_ID" \
    --function systemd-journal

Public API (assistant-facing)

Internal helpers (do NOT call directly)

These start with _ and operate on token bytes inside their own scope. They return token data through validated caller-local output variables (so the assistant never sees them on stdout). Don't shell-out to them.

Direct-agent vs Cloud-proxied: how agents_call_function chooses

Default is --via cloud -- the safe choice for any team member.

--via agent requires:

1. The agent host is reachable from the workstation on port 19999.
2. A bearer (auto-minted internally via _agents_resolve_bearer).

Falls back to --via cloud if the direct call fails.

Sensitive data

• Bearer values appear in script stderr only when masked.
• The cache file at .local/audits/.../bearers/<machine_guid>.json contains the raw bearer; mode 0600.
• Never paste bearer values, claim ids, machine GUIDs, or node UUIDs into committed files. See <repo>/.agents/sensitive-data-discipline.md for the full rule.