.agents/skills/query-agent-events/how-tos/trace-stack-symbol-regression-to-mutator.md
Use this workflow when many agent-events rows mention the same stack symbol, but the symbol may be either the faulting function or only a lower frame, and the suspected regression must be tied to a source change.
Cloud log links carry millisecond epoch bounds. Convert them to seconds for the Function payload and record the UTC interval in the local SOW:
date -u -d @"$((AFTER_MS / 1000))" '+%Y-%m-%d %H:%M:%S UTC'
date -u -d @"$((BEFORE_MS / 1000))" '+%Y-%m-%d %H:%M:%S UTC'
Do not put the Cloud link, node ID, or any event identifier in a durable artifact.
A symbol inside AE_FATAL_STACK_TRACE is not necessarily the value of AE_FATAL_FUNCTION. Use the structured crash selection first and the symbol only as a residual full-text narrower:
.agents/skills/query-agent-events/scripts/get-events.sh \
--since "$AFTER_SECONDS" \
--before "$BEFORE_SECONDS" \
--health crash \
--version all \
--query "$SYMBOL" \
--facets AE_FATAL_SIGNAL_CODE,AE_FATAL_FUNCTION,AE_AGENT_VERSION,AE_AGENT_HEALTH,AE_EXIT_CAUSE \
--last 10000 \
--output .local/audits/query-agent-events/symbol-window.json
Why:
AE_AGENT_HEALTH uses the facet index to reduce the search to crash records.query then finds the symbol anywhere in the already narrowed record, including the stack trace.--version all is intentional for a reported exact interval; auto-version filtering could hide the first affected build.Do not interpret row counts before checking the envelope:
jq '{status, partial, pagination, items, sampling:._sampling, rows:(.data|length)}' \
.local/audits/query-agent-events/symbol-window.json
Required checks:
status == 200partial == falseitems.returned == items.matcheditems.returned < items.max_to_return_sampling.sampled == 0If any check fails, narrow the query further or paginate before drawing conclusions.
Function responses store rows as arrays and provide the field indexes in columns. Always project by the column map; column order is not a stable contract.
Calculate separately:
AE_FATAL_FUNCTION == $SYMBOL;AE_FATAL_STACK_TRACE contains $SYMBOL but AE_FATAL_FUNCTION differs;The distinction matters:
AE_FATAL_FUNCTION match supports the symbol as the captured fault location.Raw stack addresses remain in the gitignored audit dump. Durable notes use only normalized frames such as 0xADDR.
The matching slice cannot provide a crash rate. Fetch all crash-class events for the affected versions over the same interval, using explicit structured version values and no FTS:
.agents/skills/query-agent-events/scripts/get-events.sh \
--since "$AFTER_SECONDS" \
--before "$BEFORE_SECONDS" \
--health crash \
--versions "$AFFECTED_VERSIONS_CSV" \
--facets AE_FATAL_SIGNAL_CODE,AE_FATAL_FUNCTION,AE_AGENT_VERSION \
--last 10000 \
--output .local/audits/query-agent-events/affected-version-crashes.json
Report at least:
Do not call per-version row shares "rates" without an active-install population denominator. Nightly rollout time strongly biases the version mix.
For a recent regression, widen to 7 days but keep both structured crash and explicit recent-version selections. Include several versions before and after the apparent boundary.
Interpretation rules:
../update-cadence.md.Map version strings to repository commits through packaging/version:
git log --format='%H %ad %s' --date=iso-strict -- packaging/version
git log -S"$VERSION" --format='%H %ad %s' --date=iso-strict -- packaging/version
Then list commits between the last unaffected and first affected nightly.
One static function can have multiple call sites with different inputs. Read every reference, then inspect the built binary's DWARF line mapping:
rg -n "\\b${SYMBOL}\\s*\\(" src
nm -an /path/to/netdata | rg "$SYMBOL|CALLER_FUNCTION"
objdump -dSl --disassemble=CALLER_FUNCTION /path/to/netdata
Match the caller line reported by the event stack to the disassembly. This can distinguish, for example, an environment-vector encoding call from a command-argument encoding call even though both enter the same function.
Use a binary built from the same source blob as the affected nightly. Verify equivalence with git rev-parse REV:path/to/source.c before relying on the mapping.
Group fault addresses without publishing them:
On glibc releases using safe-linking, a freed tcache chunk's first machine word can resemble its heap address shifted right by 12 bits. The glibc safe-linking patch defines the pointer mangling used by tcache. If the crash dereferences that shortened value from the first slot of a stale vector, this is strong use-after-free evidence.
Do not diagnose a tcache use-after-free from address shape alone. Require all of:
Search all C and C++ sources, not only *.c and *.h:
rg -n --glob '*.{c,h,cc,cpp,cxx}' \
'\b(setenv|unsetenv|putenv|clearenv|nd_setenv)\s*\(' src
git log --format='%H %ad %s' --date=iso-strict LAST_UNAFFECTED..FIRST_AFFECTED
For each candidate:
Prove it is absent from the last unaffected build and present in the first affected build:
git merge-base --is-ancestor CANDIDATE LAST_UNAFFECTED
git merge-base --is-ancestor CANDIDATE FIRST_AFFECTED
Prove its execution overlaps the crashing readers by tracing thread creation and startup order.
Check whether the mutation is common to the affected population, rather than enabled only by an uncommon optional feature.
Verify the platform contract. For example, the GNU libc environment documentation marks environment mutation as MT-Unsafe const:env; a reader walking environ concurrently is not protected by libc's writer lock. Check version-specific changes too: glibc 2.41 retained old environment arrays, reducing one failure mode without making direct concurrent environ use a portable contract.
Use this evidence ladder:
Do not skip from symptom cluster to root cause. If the final two levels are missing, label the result a working theory and state the evidence still required.
Files and guides used:
../AE_FIELDS.md../query-discipline.md../finding-crashes.md../update-cadence.md../recipes/find-by-function.mdscripts/get-events.shscripts/analyze-events.shQueries used:
jq projection through the response columns map.Validation used: