Finding fatals (deliberate exits)

A "fatal" = the agent chose to exit because of a condition it could not recover from (OOM, disk full, assertion failure, listen-socket conflict, fatal() call). DIFFERENT from a signal crash (the kernel didn't kill us; we called exit() or _exit() ourselves).

Quick recipe

.agents/skills/query-agent-events/scripts/get-events.sh \
    --exit-cause fatal \
    --since '24h ago' \
    --versions auto

--exit-cause fatal is shorthand for the deliberate-fatal class of AE_EXIT_CAUSE values (see below).

Distinguishing predicates

• AE_FATAL_SIGNAL_CODE is EMPTY (this is what separates fatals from signal crashes).
• AE_EXIT_CAUSE is one of:
  • no last status
  • out of memory (NOT cannot allocate -- the .local draft was wrong)
  • disk full / disk almost full / disk read-only
  • already running
  • fatal on start / fatal on exit / fatal and exit
  • exit timeout
  • abnormal power off
• AE_AGENT_EXIT_REASON_* may contain out-of-memory, already-running, fatal, shutdown-timeout.
• AE_FATAL_MESSAGE, AE_FATAL_FUNCTION, AE_FATAL_FILENAME, AE_FATAL_LINE -- populated from the fatal() call site.

Index-friendly query

{
  "after":  -86400,
  "before": 0,
  "last":   500,
  "__logs_sources": "agent-events",
  "selections": {
    "AE_EXIT_CAUSE": [
      "no last status",
      "out of memory",
      "disk full",
      "disk almost full",
      "disk read-only",
      "already running",
      "fatal on start",
      "fatal on exit",
      "fatal and exit"
    ],
    "AE_AGENT_VERSION": ["v2.10.0", "v2.10.0-135-nightly", "v2.10.0-130-nightly"]
  },
  "facets": ["AE_EXIT_CAUSE", "AE_FATAL_FUNCTION", "AE_HOST_ARCHITECTURE", "AE_OS_FAMILY"]
}

Per-cause triage

out of memory

Agent panicked when an allocation failed. Look at AE_FATAL_FUNCTION to localize -- often mallocz, reallocz, dbengine page allocators, or memory-pool constructors. Cross-correlate with AE_HOST_MEMORY_FREE_PERCENT (low at POST time = host was under pressure) and AE_HOST_MEMORY_NETDATA (how much netdata was using).

Most useful slicers:

• AE_AGENT_DB_MODE -- dbengine memory mode.
• AE_AGENT_DB_TIERS -- number of tiers.
• AE_HOST_MEMORY_TOTAL -- absolute host RAM. OOM on a 1GB host is different from OOM on a 64GB host.

disk full / disk almost full / disk read-only

Storage failure. Look at:

• AE_HOST_DISK_DB_FREE / AE_HOST_DISK_DB_INODES_FREE -- what state the disk was in.
• AE_HOST_DISK_DB_READ_ONLY -- bool, set on read-only.
• AE_AGENT_DB_MODE -- whether dbengine is in dbengine, ram, ram-cache, none, alloc, save, ...

already running

Listen-socket conflict at init. Look at AE_AGENT_VERSION and AE_AGENT_RESTARTS -- if restarts > 0 and the same agent_id keeps hitting this, there's a stale lock or a concurrent agent on the same host.

fatal on start / fatal on exit / fatal and exit

Generic fatal() calls. Look at AE_FATAL_MESSAGE to find the panic message and AE_FATAL_FUNCTION to localize. Group by message to find the most common patterns.

no last status

First-ever start, or the prior status file was corrupted / unreadable. Usually transient. If recurring on the same agent (AE_AGENT_RESTARTS going up), the status file is being deleted or the disk path is broken.

abnormal power off

Power loss between "agent was running" and "agent restarted". Useful to see whether power events correlate with crashes elsewhere -- usually filter these OUT of crash analysis.

exit timeout

Shutdown didn't complete in time. Look at AE_AGENT_TIMINGS_EXIT for how long shutdown ran before the timeout fired.

The timeout diagnostics are version-sensitive:

• Fixed agents store the completed per-step timing list in AE_FATAL_STACK_TRACE, beginning with shutdown timings:, and identify the hung step in AE_FATAL_MESSAGE as shutdown timed out at step: <step>.
• Older agents can overwrite the timing list during SIGABRT handling with an info: stack-trace status message. An empty AE_FATAL_MESSAGE plus info: fatal handler already captured the stack trace is the known pre-fix signature.
• AE_FATAL_FUNCTION remains shutdown_timeout in both cases, so use it as the stable grouping facet and use AE_FATAL_MESSAGE for the per-step split on fixed agents.

Triage flow

1. Get the dump -- get-events.sh --exit-cause fatal.
2. Group by exit_cause -- analyze-events.sh --by exit_cause to see the distribution.
3. For the dominant cause, group by function or message -- analyze-events.sh --by fatal_function --filter "exit_cause=out of memory".
4. Cross-check environment -- is this OOM specific to small-RAM hosts? Specific dbengine mode?
5. Read source at the panic site, fix the bug or improve the error path.

Pitfalls

• abnormal power off is not a bug -- it's environmental. Filter out of crash analysis unless you specifically want power events.

• fatal_errno vs fatal.errno -- the producer emits fatal.errno (nested), not fatal_errno (top-level). The journal field is AE_FATAL_ERRNO in both cases (the underscore comes from the dot transliteration).

• exit timeout context depends on the agent version -- use the behavior signatures above. Do not interpret the pre-fix info: fatal handler already captured the stack trace text as evidence that a real fatal stack trace was captured.

• Disk-related causes propagate -- a disk full event may show up as cannot allocate in subsequent attempts. The first record is the meaningful one.

Related recipes

• finding-crashes.md -- the OTHER class (signal-delivered).
• recipes/find-by-function.md -- localize by function name.
• recipes/find-by-version.md -- regression spotter.