Back to Netdata

AE_FIELDS reference

.agents/skills/query-agent-events/AE_FIELDS.md

2.11.021.9 KB
Original Source

AE_FIELDS reference

Verified field map for the agent-events journal namespace. Every claim here is traceable to producer source at <repo>/src/daemon/status-file.c (the schema is STATUS_FILE_VERSION = 29, src/daemon/status-file.h:14).

The .local draft agent-events-journals.md was found to have 14 high-severity divergences (wrong enums, missing fields, misattributed semantics). This file supersedes it.

How journal field names are formed

Producer JSON path -> journal field name:

  1. Producer emits a JSON path (e.g. agent.profile[0]).
  2. log2journal applies --prefix 'AE_' (literal prepend; the prefix is NOT transliterated -- src/collectors/log2journal/log2journal-help.c:108-109).
  3. log2journal walks the JSON tree. For nested objects, it joins parent + child with _. For arrays, it appends _<index> (log2journal-json.c:477-511).
  4. Per-character transliteration applies a 256-entry map (log2journal.c:8-61): lowercase -> uppercase, digits and uppercase pass through, everything else (including ., -, :, /, @, _, [, ]) maps to _. Consecutive underscores are collapsed (log2journal-json.c:395-396).

So:

  • agent.id -> AE_AGENT_ID
  • agent.profile[0] -> AE_AGENT_PROFILE_0
  • host.boot.id -> AE_HOST_BOOT_ID
  • @timestamp -> AE__TIMESTAMP (note: double underscore because @ -> _)
  • fatal.errno -> AE_FATAL_ERRNO (NOT from a top-level fatal_errno; the path is nested)

Top-level POST-time fields (always present)

Added by post_status_file() at status-file.c:967-976. These sit at the JSON top level, so they have NO nested-path prefix in the journal field name.

JSON pathJournal fieldTypeAlways?Triage meaning
exit_causeAE_EXIT_CAUSEstringyesHuman-readable label for why this session ended. The first thing to look at. See enum below.
messageAE_MESSAGEstringyesOne-line summary; subject of FTS search.
priorityAE_PRIORITYintyesSyslog priority (lower = more severe).
version_savedAE_VERSION_SAVEDuintyesThe status file's own version (typically 29).
agent_version_nowAE_AGENT_VERSION_NOWstringyesPosting agent's version (the agent that did the POST = next session, NOT the one that crashed).
agent_pid_nowAE_AGENT_PID_NOWuintyesPID of the agent that POSTed.
host_memory_criticalAE_HOST_MEMORY_CRITICALboolyesWas the host under memory pressure at POST time?
host_memory_free_percentAE_HOST_MEMORY_FREE_PERCENTuintyes% free RAM at POST time.
agent_healthAE_AGENT_HEALTHstringyesHealth classification across restart history. See enum below.
@timestampAE__TIMESTAMPRFC3339yesWhen the captured session ended.
versionAE_VERSIONuintyesThe schema version (STATUS_FILE_VERSION). Use to slice by schema.

These fields are always present on every record and are the safest scoping anchors. AE_EXIT_CAUSE, AE_AGENT_HEALTH, AE_VERSION are all good index-friendly facets.

agent.* fields (dsf_json_agent, status-file.c:99-156)

The previous (crashed) session's agent state.

JSON pathJournal fieldTypeVersion-gatingTriage meaning
agent.idAE_AGENT_IDUUIDalwaysNetdata machine GUID (per-install, persistent). The "agent identity". DIFFERENT from host.id.
agent.sinceAE_AGENT_SINCERFC3339v>=24When this install was first registered.
agent.ephemeral_idAE_AGENT_EPHEMERAL_IDUUIDalwaysUnique per Netdata invocation (changes on every restart). Use to group multiple events from the same crashed session.
agent.versionAE_AGENT_VERSIONstringalwaysThe version of the crashed session. Slice on this for regression-spotting.
agent.uptimeAE_AGENT_UPTIMEint secondsalwaysDuration the crashed session ran. Short uptime + crash = startup bug.
agent.node_idAE_AGENT_NODE_IDUUIDalwaysCloud node UUID (empty when agent isn't claimed).
agent.claim_idAE_AGENT_CLAIM_IDUUIDalwaysCloud claim UUID (empty when not claimed). Presence -> cloud-connected agent.
agent.restartsAE_AGENT_RESTARTSuintalwaysTotal restart count for this install. High value + recent crash = agent loop.
agent.crashesAE_AGENT_CRASHESuintv>=24Total crash count.
agent.pidAE_AGENT_PIDuintv>=27PID of the crashed session.
agent.postsAE_AGENT_POSTSuintv>=22Total POSTs from this install.
agent.aclkAE_AGENT_ACLKenumv>=22Cloud connection state. See enum.
agent.profile[N]AE_AGENT_PROFILE_0..Nenum arrayalwaysBitmap rendered as array. Slice by parent vs child vs iot. See enum.
agent.statusAE_AGENT_STATUSenumalwaysDAEMON_STATUS at the time of capture. See enum.
agent.exit_reason[N]AE_AGENT_EXIT_REASON_0..Nenum arrayalwaysEXIT_REASON bitmap rendered as array. Empty array = no specific reason. See enum.
agent.install_typeAE_AGENT_INSTALL_TYPEstringalwayskickstart, binpkg, static, etc. Slice on this for "is this a packaging issue?"
agent.db_modeAE_AGENT_DB_MODEstringv>=14dbengine memory mode.
agent.db_tiersAE_AGENT_DB_TIERSuintv>=14Number of dbengine tiers.
agent.kubernetesAE_AGENT_KUBERNETESboolv>=14Kubernetes deployment? Slice on this for k8s-specific issues.
agent.sentry_availableAE_AGENT_SENTRY_AVAILABLEboolv>=16Is Sentry enabled?
agent.reliabilityAE_AGENT_RELIABILITYintalwaysSigned reliability counter (positive = healthy run streak; negative = crash streak). <= -2 -> crash-loop.
agent.stack_tracesAE_AGENT_STACK_TRACESstringalwaysBacktrace backend name (libbacktrace, none).
agent.timings.initAE_AGENT_TIMINGS_INITint secondsalwaysHow long startup took. Long init + crash = startup bug.
agent.timings.exitAE_AGENT_TIMINGS_EXITint secondsalwaysHow long shutdown took.

metrics.* fields (dsf_json_metrics, :158-193)

Snapshot of the database at the time of capture. Useful for "big-database crashes" investigations.

JSON pathJournal field
metrics.nodes.totalAE_METRICS_NODES_TOTAL
metrics.nodes.receivingAE_METRICS_NODES_RECEIVING
metrics.nodes.sendingAE_METRICS_NODES_SENDING
metrics.nodes.archivedAE_METRICS_NODES_ARCHIVED
metrics.metrics.collectedAE_METRICS_METRICS_COLLECTED
metrics.metrics.availableAE_METRICS_METRICS_AVAILABLE
metrics.instances.collectedAE_METRICS_INSTANCES_COLLECTED
metrics.instances.availableAE_METRICS_INSTANCES_AVAILABLE
metrics.contexts.collectedAE_METRICS_CONTEXTS_COLLECTED
metrics.contexts.availableAE_METRICS_CONTEXTS_AVAILABLE

host.* fields (dsf_json_host, :195-253)

Host-level info, mostly stable across crashes on the same host.

JSON pathJournal fieldTriage meaning
host.idAE_HOST_IDOS-level /etc/machine-id. Different from AE_AGENT_ID (Netdata's own identifier). Use to spot multiple agents on the same host.
host.architectureAE_HOST_ARCHITECTUREx86_64, aarch64, armv7l, ... Slice for arch-specific bugs.
host.virtualizationAE_HOST_VIRTUALIZATIONnone, kvm, vmware, lxc, docker, ...
host.containerAE_HOST_CONTAINERnone, docker, kubernetes, ...
host.uptimeAE_HOST_UPTIMEMISLEADING NAME. Stores boottime EPOCH (status-file.c:202 writes ds->boottime from now_boottime_sec()). NOT a duration. Compute uptime via now - AE_HOST_UPTIME.
host.timezoneAE_HOST_TIMEZONEstring, v>=20
host.cloud_providerAE_HOST_CLOUD_PROVIDERaws, gcp, azure, ..., v>=20
host.cloud_instanceAE_HOST_CLOUD_INSTANCEEC2 instance type etc., v>=20
host.cloud_regionAE_HOST_CLOUD_REGIONv>=20
host.system_cpusAE_HOST_SYSTEM_CPUSuint. Slice for "low-cpu environment" bugs.
host.boot.idAE_HOST_BOOT_IDUUID, changes on every host boot.
host.memory.totalAE_HOST_MEMORY_TOTALbytes, only when OS_SYSTEM_MEMORY_OK
host.memory.freeAE_HOST_MEMORY_FREEbytes
host.memory.netdataAE_HOST_MEMORY_NETDATAbytes used by netdata, v>=21
host.memory.oom_protectionAE_HOST_MEMORY_OOM_PROTECTIONuint, v>=21
host.disk.db.totalAE_HOST_DISK_DB_TOTALbytes available to dbengine
host.disk.db.freeAE_HOST_DISK_DB_FREEbytes free
host.disk.db.inodes_totalAE_HOST_DISK_DB_INODES_TOTALuint
host.disk.db.inodes_freeAE_HOST_DISK_DB_INODES_FREEuint
host.disk.db.read_onlyAE_HOST_DISK_DB_READ_ONLYbool. True + crash -> "disk read-only" cause.
host.disk.netdata.dbengineAE_HOST_DISK_NETDATA_DBENGINEbytes used by dbengine files.
host.disk.netdata.sqliteAE_HOST_DISK_NETDATA_SQLITEbytes used by SQLite files.
host.disk.netdata.otherAE_HOST_DISK_NETDATA_OTHERbytes used by other files.
host.disk.netdata.last_updatedAE_HOST_DISK_NETDATA_LAST_UPDATEDRFC3339.

os.* fields (dsf_json_os, :255-266)

JSON pathJournal fieldTriage meaning
os.typeAE_OS_TYPEenum: unknown, linux, freebsd, macos, windows.
os.kernelAE_OS_KERNELKernel version string.
os.nameAE_OS_NAMEDistro name (e.g. Ubuntu, CentOS Stream).
os.versionAE_OS_VERSIONDistro version.
os.familyAE_OS_FAMILYos_id (e.g. ubuntu). Slice for distro-specific issues.
os.platformAE_OS_PLATFORMos_id_like (parent distro family, e.g. debian). NOT a rewrite of AE_OS_FAMILY -- they're independent producer fields.

hw.* fields (dsf_json_hw, :268-319)

DMI / SMBIOS data. Useful for hardware-specific bug investigation. Privacy-sensitive serials and asset_tags are commented out at the producer side (status-file.c:275-276, :294-295, :304-305) and never reach the journal.

JSON pathJournal fieldNotes
hw.sys.vendorAE_HW_SYS_VENDORBIOS / system vendor.
hw.sys.uuidAE_HW_SYS_UUIDSystem UUID.
hw.product.idAE_HW_PRODUCT_IDRaw platform product/model identifier (e.g. Mac16,10).
hw.product.nameAE_HW_PRODUCT_NAMEHuman-friendly product name when available (e.g. Mac mini (M4, 2024)).
hw.product.versionAE_HW_PRODUCT_VERSION
hw.product.skuAE_HW_PRODUCT_SKU
hw.product.familyAE_HW_PRODUCT_FAMILY
hw.board.nameAE_HW_BOARD_NAME
hw.board.versionAE_HW_BOARD_VERSION
hw.board.vendorAE_HW_BOARD_VENDOR
hw.chassis.typeAE_HW_CHASSIS_TYPENumeric (e.g. 6 = desktop, 9 = laptop).
hw.chassis.vendorAE_HW_CHASSIS_VENDOR
hw.chassis.versionAE_HW_CHASSIS_VERSION
hw.bios.dateAE_HW_BIOS_DATE
hw.bios.releaseAE_HW_BIOS_RELEASE
hw.bios.versionAE_HW_BIOS_VERSION
hw.bios.vendorAE_HW_BIOS_VENDOR

product.* fields (dsf_json_product, :321-330)

JSON pathJournal field
product.vendorAE_PRODUCT_VENDOR
product.idAE_PRODUCT_ID
product.nameAE_PRODUCT_NAME
product.typeAE_PRODUCT_TYPE

fatal.* fields (dsf_json_fatal, :331-367)

Present on crashes and deliberate fatal conditions. Empty on graceful exits.

JSON pathJournal fieldVersion-gatingTriage meaning
fatal.lineAE_FATAL_LINEalwaysSource line of the panic. Combine with FILENAME and FUNCTION for de-dup.
fatal.filenameAE_FATAL_FILENAMEalwaysSource file. Slice on this for "this file is buggy".
fatal.functionAE_FATAL_FUNCTIONalwaysFunction name (with demangled symbol). Slice on this for "this function is buggy".
fatal.messageAE_FATAL_MESSAGEalwaysPanic message. Subject of FTS.
fatal.errnoAE_FATAL_ERRNOalwayserrno string at panic.
fatal.threadAE_FATAL_THREADalwaysWorker thread name (e.g. CTXLOAD, STREAM:63).
fatal.thread_idAE_FATAL_THREAD_IDalwaysPOSIX TID.
fatal.stack_traceAE_FATAL_STACK_TRACEalwaysBacktrace. Real addresses preserved (anonymization is dedup-only, status-file-dedup.c:26-36).
fatal.signal_codeAE_FATAL_SIGNAL_CODEv>=16SIGNAL/SI_CODE formatted (e.g. SIGSEGV/SEGV_MAPERR). Empty -> not a signal crash. Primary signal-crash predicate. See enum.
fatal.sentryAE_FATAL_SENTRYv>=17Was a Sentry submission attempted?
fatal.fault_addressAE_FATAL_FAULT_ADDRESSv>=18Hex address of the fault. Empty when signal_code == 0.
fatal.worker_job_idAE_FATAL_WORKER_JOB_IDv>=23Worker job ID at panic.

Enum reference

AE_AGENT_STATUS (DAEMON_STATUS)

Source: src/daemon/status-file.c:23-33.

ValueMeaning for triage
noneNo prior status (very first session).
initializingCrashed during startup -> startup bug. Combine with agent.timings.init for context.
runningCrashed during normal operation -> the most "interesting" class.
exitingCrashed during shutdown -> shutdown-path bug.
exitedGraceful exit (no crash).

AE_AGENT_ACLK (CLOUD_STATUS)

Source: src/claim/cloud-status.c:5-15.

ValueMeaning for triage
availableDefault; not yet attempted.
onlineConnected to Cloud (ACLK up).
indirectConnected via parent.
bannedCloud rejected (claim issue).
offlineDisconnected (network or shutdown).

(The .local draft listed a disabled value -- it does NOT exist in the producer source.)

AE_AGENT_HEALTH

Source: src/daemon/status-file.c:929-952. Computed by the agent (not the ingestion server) at POST time across restart history. Used to isolate crash classes.

ValueMeaning for triage
healthy-firstFirst run, no prior crashes. Boring (filter out).
healthy-repeatedMultiple healthy runs in a row.
healthy-loopReliability >= 2 consecutive healthy runs.
healthy-recoveredWas unhealthy, now healthy.
crash-firstFirst crash ever on this install. Interesting -- new bug?
crash-enteredSingle crash, then recovered.
crash-loopReliability <= -2 (repeated crashes). Highest-priority class.
crash-repeatedTwo or more crashes.

To find ALL crashes: (AE_AGENT_HEALTH in crash-first, crash-loop, crash-repeated, crash-entered).

AE_AGENT_PROFILE_* (ND_PROFILE bitmap)

Source: src/daemon/config/netdata-conf-profile.c:7-15.

ValueMeaning
standaloneSingle-node deployment.
parentStreaming parent. Slice for "parent-only" bugs.
childStreaming child. Slice for "child-only" bugs.
iotIoT / lightweight profile.

(The .local draft listed dopple and store-child -- they do NOT exist; iot was missing.)

AE_AGENT_EXIT_REASON_* (EXIT_REASON bitmap)

Source: src/libnetdata/exit/exit_initiated.c:7-38. The EXIT_REASON bitmap renders as a JSON array. Empty bitmap -> empty array (no none element).

20 distinct strings:

ValueMeaning
signal-segmentation-faultSIGSEGV received.
signal-bus-errorSIGBUS received.
signal-floating-point-exceptionSIGFPE received.
signal-illegal-instructionSIGILL received.
signal-abortSIGABRT received (assertion / abort()).
signal-bad-system-callSIGSYS received.
signal-cpu-time-limit-exceededSIGXCPU received.
signal-file-size-limit-exceededSIGXFSZ received.
signal-quitSIGQUIT received.
signal-terminateSIGTERM received (graceful kill).
signal-interruptSIGINT received (Ctrl-C).
out-of-memoryOOM panic.
already-runningAnother instance held the listen socket.
fatalGeneric fatal() call.
api-quitAPI endpoint requested exit.
cmd-exitExplicit netdata --exit invocation.
service-stopService manager (systemd) sent stop.
system-shutdownHost shutting down.
updateReplaced by a new version.
shutdown-timeoutShutdown took too long.

(The .local draft was significantly wrong here -- listed ~10 invented values like exit-called, exit-and-update, cannot-allocate, oom, assertion-failed, none of which exist in producer source.)

AE_EXIT_CAUSE (top-level)

Source: src/daemon/status-file.c:1097-1286. Computed by the agent, NOT the ingestion server. The most useful field for classifying records.

26 distinct strings:

Initial / no prior state (1):

ValueMeaning
no last statusFirst-ever start; no prior status file readable.

Prior was EXITED (graceful) (7):

ValueMeaning
exit no reasonPrior exited cleanly with no reason recorded.
deadly signal and exitGot a deadly signal but exited normally.
fatal and exitHit a fatal but managed to exit.
exit on system shutdownHost shutting down; agent stopped gracefully.
exit to updateStopped to allow an update.
exit and updatedStopped and was replaced by a new version.
exit instructednetdata --exit or service stop.

Prior was INITIALIZING (8):

ValueMeaning
abnormal power offPower loss during startup.
deadly signal on startSignal during startup.
out of memoryOOM during startup. (.local draft says cannot allocate -- wrong.)
already runningListen socket conflict at init.
disk read-onlyFilesystem read-only at init.
disk fullDisk full at init.
disk almost fullDisk near capacity at init.
fatal on startfatal() during startup.
killed hard on startSIGKILL/SIGTERM during startup.

Prior was EXITING (5):

ValueMeaning
deadly signal on exitSignal during shutdown.
exit timeoutShutdown didn't complete in time.
fatal on exitfatal() during shutdown.
killed hard on shutdownSIGKILL during shutdown (host shutdown).
killed hard on updateSIGKILL during hot update.
killed hard on exitSIGKILL during exit.

Prior was RUNNING (6):

ValueMeaning
abnormal power offPower loss during normal operation.
out of memoryOOM during normal operation.
deadly signalSignal received during normal operation.
killed fatalSIGKILL after a fatal.
killed hard low ramOOM-killed (RAM pressure).
killed hardSIGKILL/SIGTERM from outside.

AE_OS_TYPE (DAEMON_OS_TYPE)

Source: src/daemon/status-file.c:35-45.

unknown, linux, freebsd, macos, windows.

AE_FATAL_SIGNAL_CODE

Format: SIGNAL/SI_CODE (e.g. SIGSEGV/SEGV_MAPERR). Sources: src/libnetdata/signals/signal-code.c:12-53 (signal name map), :97-184 (per-signal SI_CODE map).

Most relevant for crash triage:

ValueMeaning
SIGSEGV/SEGV_MAPERRInvalid memory map (NULL pointer, freed memory).
SIGSEGV/SEGV_ACCERRAccess violation (write to read-only page).
SIGSEGV/SEGV_BNDERRAddress bound check fault.
SIGSEGV/SEGV_PKUERRProtection key fault.
SIGBUS/BUS_ADRALNAlignment error.
SIGBUS/BUS_ADRERRNon-existent physical address.
SIGBUS/BUS_OBJERRObject-specific bus error.
SIGFPE/FPE_INTDIVInteger divide by zero.
SIGFPE/FPE_INTOVFInteger overflow.
SIGFPE/FPE_FLTDIVFloat divide by zero.
SIGABRT/SI_TKILLabort() / assertion failure (typical SI_CODE for abort()).
SIGTRAP/TRAP_BRKPTBreakpoint trap.
SIGTRAP/TRAP_TRACETrace trap.

Empty AE_FATAL_SIGNAL_CODE -> not a signal crash (it's a deliberate fatal or a graceful exit).

(The .local draft had SIGABRT/ABRT which is wrong: ABRT is not a valid SI_CODE token. And SIGTRAP/TRAP Trace should be SIGTRAP/TRAP_TRACE. And SIGVTALRM/VTALRM does not exist as a per-signal SI_CODE.)

Index-friendly facets (high-value)

These fields are the first-pass slicers for queries. Always include at least 1-2 of these in selections before falling back to FTS:

  • AE_AGENT_VERSION -- regression / fix-detection.
  • AE_AGENT_HEALTH -- crash class.
  • AE_EXIT_CAUSE -- exit class.
  • AE_FATAL_SIGNAL_CODE -- signal type.
  • AE_FATAL_FUNCTION -- localize to a function.
  • AE_FATAL_FILENAME -- localize to a file.
  • AE_HOST_ARCHITECTURE -- arch-specific bugs.
  • AE_OS_FAMILY -- distro-specific bugs.
  • AE_AGENT_PROFILE_0 (and _1, _2) -- parent / child / iot.
  • AE_AGENT_KUBERNETES -- k8s-specific.
  • AE_AGENT_INSTALL_TYPE -- packaging-specific.

Privacy-sensitive fields

Treat these as identifying. The redact-events.sh opt-in filter masks them when sharing:

  • AE_AGENT_ID (machine GUID).
  • AE_HOST_ID (OS machine-id).
  • AE_AGENT_NODE_ID, AE_AGENT_CLAIM_ID (Cloud identifiers).
  • AE_HOST_BOOT_ID, AE_AGENT_EPHEMERAL_ID.
  • AE_HW_SYS_UUID.
  • DMI fields (AE_HW_*) when correlated with serial-equivalent identifiers.

(Privacy-sensitive serials and asset_tags are already commented out at the producer side and never reach the journal -- status-file.c:275-276, :294-295, :304-305.)

What is NOT in the journal

Several producer fields are intentionally redacted at the producer side (commented out in dsf_json_hw):

  • hw.sys.serial, hw.sys.asset_tag
  • hw.board.serial, hw.board.asset_tag
  • hw.chassis.serial, hw.chassis.asset_tag

There is no agent.happiness field in the producer source at any version. The .local draft mentioned it -- the field has never existed.

Stack trace addresses are NOT anonymized in the journal

status-file-dedup.c:26-36 zeroes out hex addresses ONLY when computing the dedup hash. The journal-emitted AE_FATAL_STACK_TRACE retains real addresses. Useful for bug investigation; sensitive when sharing externally.