SNMP Trap Metric Profiles

Status

• Status: Implemented. The current contract uses explicit per-job rule selection, canonical profile YAML, fixed identity/cardinality/overflow policy, and fixed TTL clear-and-expire behavior.
• Design status: revised before stable release on 2026-08-01 to remove speculative selection modes, noncanonical aliases, and configurable policy branches. The authoritative operator schema is profile-format.md.
• Delivery history:
  • Phase 1: external gap analysis of the use-case inventory.
  • Phase 2: design proposal and external design review.
  • Implementation Phase A: profile-defined trap metrics.
  • Implementation Phase B: receiver/pipeline metrics.

Purpose

Netdata needs a trap-to-metrics system that lets operators convert selected SNMP trap signals into useful metrics without losing the forensic value of trap logs.

The system must support practical, profile-backed use cases across stock and operator-provided trap profiles. The first requirement is to identify the whole goal before designing the schema.

Scope Of This Spec

This document records:

• operator use cases;
• concrete trap/profile examples;
• the design constraints that shaped the implementation;
• the canonical profile-based contract for supporting these use cases.

This spec intentionally does not define:

• runtime data structures;
• exact Go types;
• file-by-file implementation plan;
• final default enablement policy for stock-generated metric rules;
• UI or dashboard design.
• exact Go types for receiver/pipeline metrics. This spec defines the product-level requirements those metrics must satisfy when implemented as Phase B.

Ground Rules

• Per-device visibility is mandatory. A trap listener often receives traps from many devices, so totals across all devices are not enough.
• Trap logs remain the source of full event detail. Metrics are derived signals for dashboards, alerting, trend detection, and quick triage.
• High-cardinality values such as usernames, MAC addresses, source IPs, interface names, peer addresses, alert keys, and free-form descriptions must not become unbounded metric labels by default.
• A trap-derived state is event-driven and lossy unless reconciled by polling. Missed clear traps, packet loss, and device reboot can make state stale.
• The stock profile pack can provide common, safe behavior, but operators must be able to add or adjust site-specific trap metrics through custom profiles.
• Every accepted use case must be backed by real profile/MIB evidence or by an established open-source monitoring implementation.

Industry Context

Existing open-source NMS implementations mostly treat SNMP traps as events, alarms, or state mutations, not as a profile-driven trap-to-time-series extraction system. Netdata should reuse their operational lessons for resource identity, problem/clear pairing, and cardinality, but should not assume there is a mature open-source trap-metric profile schema to copy.

Evidence:

• librenms/librenms @ 5ffbb16324ad7c25f9588801ca9d4d52da45ea21 LibreNMS/Snmptrap/Handlers/LinkDown.php:48 through :74 resolves ifIndex, updates the matched port, and logs interface-scoped events.
• librenms/librenms @ 5ffbb16324ad7c25f9588801ca9d4d52da45ea21 LibreNMS/Snmptrap/Handlers/BgpEstablished.php:48 through :65 resolves a BGP peer from the trap OID suffix and updates peer state.
• opennms/opennms @ 40cc8535351f09c24978771a8832cfc286b85572 docs/modules/operation/pages/deep-dive/alarms/configuring-alarms.adoc:56 through :142 documents reduction keys, problem/resolution alarm types, and clear keys.

Pre-Phase-A Implementation Baseline

Historical facts from the unreleased SNMP trap collector before Phase A replaced job-level operator metrics with profile-defined metrics:

• Job-level operator metrics were configured as oid, context, and optional dimension_from_varbind.
• The old runtime incremented counters when the trap OID matched.
• The old runtime did not extract numeric values from trap varbinds.
• The old runtime did not filter metrics by varbind predicates.
• The old runtime rejected duplicate configured OIDs, so one trap OID could not emit several independent metrics through the job-level contract.
• Existing static metric instances were scoped by listener job, not by source device.
• Trap profile YAML had varbinds and traps, but no profile-local metric section.

Evidence:

• Pre-Phase-A src/go/plugin/go.d/collector/snmp_traps/config.go:72 through :76.
• Pre-Phase-A src/go/plugin/go.d/collector/snmp_traps/operator_metric.go:23 through :55 (file removed by Phase A).
• Pre-Phase-A src/go/plugin/go.d/collector/snmp_traps/operator_metric.go:192 through :219 (file removed by Phase A).
• Pre-Phase-A src/go/plugin/go.d/collector/snmp_traps/operator_metric.go:380 through :420 (file removed by Phase A).
• Pre-Phase-A src/go/plugin/go.d/collector/snmp_traps/profile.go:156 through :164.

Use Case 1: Per-Device Trap Activity

Operator question:

• Which device is producing trap activity?
• Which devices suddenly started sending more state-change, security, auth, config-change, or diagnostic traps?
• Which device is responsible for a spike seen on a shared trap listener?

Concrete examples:

• A single trap listener receives IF-MIB::linkDown and IF-MIB::linkUp from many switches.
• A single trap listener receives SNMPv2-MIB::authenticationFailure from many devices.
• A trap hub receives BGP transition traps from many routers.

Why it matters:

• Listener-level totals are operationally ambiguous. They answer "did the hub receive traps?", not "which device is affected?".
• Device-level views are the minimum useful identity for Netdata dashboards, alerts, and triage.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:15536 through :15555 defines IF-MIB::linkDown and IF-MIB::linkUp.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:15557 defines SNMPv2-MIB::authenticationFailure.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:10068 through :10086 includes current BGP established/backward-transition notifications.

Use Case 2: Filtered Event Counters

Operator question:

• How many trap events of a specific kind happened after excluding expected or low-value events?
• Can planned or harmless state changes be separated from operational problems?

Concrete examples:

• Count IF-MIB::linkDown only when ifAdminStatus=up, so planned administrative shutdowns are not counted as unexpected link failures.
• Count Cisco CLI running-config changes only for terminal types such as console, terminal, virtual, or another explicitly selected enum value.
• Count only CloudVision firing notifications whose carried severity is critical.

Why it matters:

• Trap OID alone is often too coarse.
• Profiles already know the varbind names and many enum values needed to make useful distinctions.
• Without filtering, operators must either count noisy events or avoid metrics for those traps entirely.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:2209 through :2215 defines ifAdminStatus enum values.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:15536 through :15545 shows linkDown carries ifAdminStatus.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:4211 through :4220 defines Cisco terminal type enum values.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:4221 through :4224 defines the terminal user as DisplayString; this is useful log detail, not a safe default metric label.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:33139 through :33148 shows ccmCLIRunningConfigChanged carries terminal type and terminal user.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/arista-networks-inc-formerly-arastra-inc.yaml:106 through :113 defines aristaCvAlertSeverity.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/arista-networks-inc-formerly-arastra-inc.yaml:271 through :284 shows CloudVision firing notifications carry the severity varbind.

Use Case 3: Numeric Samples Carried By Traps

Operator question:

• What value did the device report when it emitted the trap?
• How close was the resource to the threshold or limit?
• Did the reported value trend upward across repeated threshold traps?

Concrete examples:

• Arista hardware utilization alerts carry in-use entries and high-watermark values.
• Juniper DFC packet-rate threshold traps carry input packet rate and watermark values.
• Juniper IDP CPU and memory notifications carry usage and threshold values.

Why it matters:

• Some traps are not just event notifications. They carry the measurement that triggered the notification.
• Counting those traps loses the most useful numeric information.
• Operators need docs to understand that these are last trap-reported values held by the collector, not continuously polled gauges.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/arista-networks-inc-formerly-arastra-inc.yaml:244 through :253 shows Arista hardware utilization values.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/juniper-networks-inc.yaml:4428 through :4460 shows Juniper DFC packet-rate and watermark values.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/juniper-networks-inc.yaml:4542 through :4565 shows Juniper IDP CPU/memory usage and threshold values.

Use Case 4: Multiple Metrics From One Notification Or Notification Family

Operator question:

• Can one received trap update all useful metric values carried by that trap?
• Can related reported values be charted together without requiring duplicate trap-profile copies?

Concrete examples:

• Arista CLB flow threshold notifications carry allocated, unallocated, learned, threshold, and limit values across the same notification family.
• Juniper DFC memory threshold traps carry flow usage, flow watermarks, criteria usage, and criteria watermarks in one notification.

Why it matters:

• Real traps frequently carry several related numeric fields.
• Restricting one trap OID to one metric forces operators to choose one value and discard the rest.
• Related values often belong on the same chart or in a small group of charts.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/arista-networks-inc-formerly-arastra-inc.yaml:300 through :347 shows Arista CLB flow fields.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/juniper-networks-inc.yaml:4497 through :4523 shows Juniper DFC hard memory fields.
• Pre-Phase-A src/go/plugin/go.d/collector/snmp_traps/operator_metric.go:27 through :41 shows the removed job-level metrics rejected duplicate OIDs.

Use Case 5: Per-Resource Trap Metrics

Operator question:

• Which interface, VLAN, service set, port group, sensor, peer, or local device resource is affected?
• Can the dashboard show the noisy or failing resource without turning every arbitrary varbind into a metric label?

Concrete examples:

• Count linkDown and linkUp per device interface using ifIndex or an enriched interface identity.
• Count Cisco port-security violations per interface or VLAN while keeping the violating MAC address in logs only.
• Track Juniper DFC packet-rate threshold events per interface.
• Track Arista CLB flow threshold events per port group.

Why it matters:

• Device-level identity is necessary but not always sufficient.
• Network operators often troubleshoot at interface, peer, sensor, or service object level.
• Raw resource names can be high-cardinality, unstable, or sensitive, so the use case requires bounded resource identity rather than arbitrary labels.
• Resource strings such as port-group names can be used only when the profile or operator explicitly treats them as bounded resource identities.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:2224 through :2227 defines ifIndex as a large interface index range.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:37706 through :37735 shows Cisco port-security traps carry interface/VLAN/MAC detail.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/juniper-networks-inc.yaml:4428 through :4460 shows Juniper DFC packet-rate traps carry interface names.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/arista-networks-inc-formerly-arastra-inc.yaml:300 through :347 shows Arista CLB port-group fields.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/profile-format.md:337 through :342 distinguishes high-cardinality log content from bounded metric labels.

Use Case 6: Trap-Derived Current State

Operator question:

• Is this resource currently in a problem state according to the last received trap pair?
• Which resources are currently firing, asserted, exceeded, down, or degraded?

Concrete examples:

• IF-MIB::linkDown marks an interface down; matching IF-MIB::linkUp clears it.
• Arista external alarm asserted/deasserted notifications track active external alarms.
• Arista CloudVision firing/resolved notifications track active CloudVision alerts.
• Juniper exceeded/under-threshold or notify/restored pairs track active threshold state.

Why it matters:

• Event counters show rate and flapping, but they do not answer "is it still active?".
• Many vendor MIBs model problem and clear notifications as separate traps.
• The resulting state is useful, but it must be documented as trap-derived and potentially stale when clear traps are missed.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:15536 through :15555 defines linkDown and linkUp.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/arista-networks-inc-formerly-arastra-inc.yaml:254 through :269 defines external alarm asserted/deasserted notifications.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/arista-networks-inc-formerly-arastra-inc.yaml:271 through :299 defines CloudVision firing/resolved notifications.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/juniper-networks-inc.yaml:4428 through :4523 includes exceeded/under-threshold notification pairs.

Use Case 7: Severity-Aware Metric Filtering And Grouping

Operator question:

• Can metrics separate critical vendor alerts from lower-priority vendor alerts?
• Can vendor-specific severity values be normalized for metric grouping or filtering while the raw value remains available in logs?

Concrete examples:

• Arista CloudVision firing/resolved notifications carry aristaCvAlertSeverity with values info, warning, error, and critical.
• A user may want only critical CloudVision firing notifications to feed a critical-alert metric, while retaining all CloudVision trap logs.

Why it matters:

• Static profile severity describes the default Netdata classification for the trap OID.
• Some vendors put the actual alert severity in a varbind, so the same trap OID can carry different operational priority.
• Metrics may need severity-aware filtering or grouping without changing the journal severity contract in this phase.
• This is a specialization of filtered counters and trap-derived state, not a separate metric kind.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/arista-networks-inc-formerly-arastra-inc.yaml:106 through :113 defines aristaCvAlertSeverity.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/arista-networks-inc-formerly-arastra-inc.yaml:271 through :299 shows CloudVision firing/resolved traps carry severity.

Use Case 8: Audit And Security Counters With Sensitive Detail In Logs

Operator question:

• How often are configuration changes, auth failures, and security violations happening per device or resource?
• Can sensitive or high-cardinality event detail remain searchable in logs while metrics stay bounded?

Concrete examples:

• Cisco configuration-change traps include terminal user and terminal type.
• Cisco port-security traps include interface, VLAN, and violating MAC address.
• SNMP authentication-failure traps indicate authentication problems, but the system must not expose credentials or sensitive source details as metric labels.

Why it matters:

• Security and audit workflows need counters and trends, but the full event details are often sensitive or high-cardinality.
• Metrics should answer rate and scope questions; logs should preserve the full event context.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:33139 through :33148 shows Cisco config-change user and terminal varbinds.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:37706 through :37735 shows Cisco port-security MAC/interface/VLAN detail.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:23357 through :23360 defines the last secure MAC address as MacAddress; this is useful log detail, not a safe default metric label.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:15557 defines SNMPv2-MIB::authenticationFailure.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/profile-format.md:337 through :342 records the existing bounded-label rule.

Use Case 9: Lifecycle And Availability Event Counters

Operator question:

• Which devices rebooted or restarted their SNMP entity?
• Can lifecycle changes be tracked per device without turning each event into a long-lived state metric?

Concrete examples:

• SNMPv2-MIB::coldStart and SNMPv2-MIB::warmStart count device SNMP entity initialization events.
• Vendor restart or management-agent restart traps can be counted the same way when profiles identify them.

Why it matters:

• Some traps are best treated as event history and rates, not numeric samples or current state.
• These counters are useful for flapping detection, incident timelines, and post-reboot verification.
• Routing and adjacency churn is related, but it is a separate use case because it needs per-peer identity.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:15525 through :15534 defines coldStart and warmStart.

Use Case 10: Operator-Defined Custom Trap Semantics

Operator question:

• How can an operator add metrics for vendor, site, or product traps that Netdata does not ship yet?
• How can an operator define metrics for generic or user-defined trap slots whose semantics are site-specific?

Concrete examples:

• A vendor MIB has local user-defined event traps.
• A site uses custom enterprise traps for UPS, environmental, access-control, or application events.
• A stock profile decodes a trap, but the operator wants a site-specific metric view or threshold filter.

Why it matters:

• Stock profiles cannot predict every site-specific trap meaning.
• Operators need a supported path that survives package upgrades.
• Custom metric behavior should live with custom trap-profile knowledge, not in collector source code.
• This is a cross-cutting authoring requirement. It does not add a new metric behavior by itself; it applies to every use case in this document.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/profile-format.md:36 through :46 documents stock and operator trap profile directories.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/profile-format.md:132 through :137 documents converting custom MIBs offline and dropping YAML files under /etc/netdata/go.d/snmp.trap-profiles/.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/profile-format.md documents complete same-identity replacement and independent operator files.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/milestone-systems-a-s.yaml:17 shows a shipped profile containing a user-defined event trap example.

Use Case 11: Standard Alarm Set/Clear Metrics

Operator question:

• Which standard SNMP alarms are currently set per device or resource?
• Which alarm classes, probable causes, and perceived severities are recurring?
• Can a single status-change trap OID both assert and clear a state metric based on a carried condition varbind?

Concrete examples:

• SNMP-ALARM-MIB::snmpAlarmStatusChange carries snmpAlarmLogCond with set or clear values plus an alarm identifier.
• SNMP-ALARM-MIB::snmpItuAlarmStatusChange carries ITU alarm class, probable cause, perceived severity, and additional text.
• RMON-MIB::risingAlarm and RMON-MIB::fallingAlarm carry the sampled value, threshold, sample type, and alarmed variable.

Why it matters:

• Some common alarm MIBs do not use separate problem and clear trap OIDs.
• Trap-to-state logic must support set/clear semantics carried by a varbind, not only OID pairs.
• Alarm identifiers and affected variables can be resource keys, but additional text must remain log detail unless explicitly bounded.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:5516 through :5525 defines snmpAlarmLogCond set/clear values and snmpAlarmLogId.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:5559 through :5588 defines ITU alarm class, severity, and probable-cause fields.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:10067 through :10128 defines BGP and RMON rising/falling notifications.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:15563 through :15590 defines SNMP-ALARM-MIB status-change notifications and their varbinds.
• opennms/opennms @ 40cc8535351f09c24978771a8832cfc286b85572 docs/modules/operation/pages/deep-dive/alarms/configuring-alarms.adoc:94 through :134 documents problem/resolution alarms and clear keys.

Use Case 12: Environmental, Power, And Component State

Operator question:

• Which device sensor, fan, power supply, UPS, battery, or environmental contact is in a problem state?
• What value did the device report for voltage, temperature, CPU, or another component measurement when the trap fired?
• Which problem/clear pairs are flapping?

Concrete examples:

• APC UPS traps report overload, on-battery, low-battery, battery-needs-replacement, fault, and corresponding clear events.
• Cisco ENVMON traps report voltage, temperature, fan, and power-supply state changes, with value varbinds for voltage and temperature.
• Cisco CPU threshold traps report threshold and current interval values.

Why it matters:

• Trap profiles contain many hardware and environmental notifications that are operationally closer to component health than generic event counts.
• Operators need per-device and, where bounded, per-component visibility.
• Numeric values carried by these traps are useful samples, while problem/clear pairs are useful trap-derived state.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/american-power-conversion-corp.yaml:409 through :449 defines UPS overload, on-battery, and low-battery traps.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/american-power-conversion-corp.yaml:481 through :611 defines low-battery clear, battery replacement, contact fault, fan failure, and battery-pack communication events.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/american-power-conversion-corp.yaml:660 through :684 defines overload-cleared and battery-replaced traps.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:32724 through :32781 defines Cisco voltage and temperature notifications with value and state varbinds.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:32745 through :32761 defines Cisco fan and redundant power-supply notifications.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:33716 through :33736 defines Cisco CPU rising/falling threshold notifications.

Use Case 13: Routing Protocol, HA, And Adjacency State

Operator question:

• Which peer, neighbor, or redundancy group changed state?
• Which device has routing adjacency churn or HA state flapping?
• Is a routing or HA session currently up, down, degraded, or in a vendor state according to the last trap?

Concrete examples:

• Standard BGP established and backward-transition notifications carry the BGP peer address and peer state.
• Vendor OSPF neighbor state-change notifications carry neighbor identity and neighbor state.
• Cisco HSRP notifications carry standby group state.

Why it matters:

• Device lifecycle counters are not enough for routing and HA operations.
• The useful identity is usually device plus peer, neighbor, or group, not only the trap OID.
• Peer addresses and neighbor identifiers can explode cardinality in large networks, so profiles need explicit resource-label discipline.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:10068 through :10086 defines current BGP established/backward-transition notifications with bgpPeerRemoteAddr and bgpPeerState.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/nbase-switch-communication.yaml:2757 through :2769 defines OSPF neighbor state-change varbinds.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:33708 through :33714 defines Cisco HSRP state-change varbinds.
• librenms/librenms @ 5ffbb16324ad7c25f9588801ca9d4d52da45ea21 LibreNMS/Snmptrap/Handlers/BgpBackwardTransition.php:48 through :66 resolves a BGP peer and updates peer state.

Use Case 14: Capacity, Pool, And Utilization Thresholds

Operator question:

• Which device or bounded resource crossed a capacity threshold?
• What was the reported utilization, pool count, or allocation count at the time?
• Did the threshold condition clear?

Concrete examples:

• Cisco DHCP free-address low/high threshold traps carry scope, current free address value, threshold type, and threshold value.
• Cisco CGN NAT port-usage watermark traps carry current port allocation and low or high threshold values.
• Cisco CPU and RMON rising/falling traps carry reported values and thresholds.

Why it matters:

• Capacity traps are often only useful when the reported numeric value and the configured threshold are retained as metrics.
• Many capacity conditions have a clear notification and can also drive trap-derived state.
• Scope names and variables need bounded-resource handling; raw names are not safe labels by default.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:34109 through :34133 defines Cisco DHCP free-address low/high threshold traps and carried scope/value/threshold varbinds.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:45656 through :45691 defines Cisco CGN port-usage low/high watermark and clear traps.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/ciscosystems.yaml:33716 through :33736 defines Cisco CPU threshold value and current interval value varbinds.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/standard.yaml:10105 through :10128 defines RMON rising/falling alarm value and threshold varbinds.

Use Case 15: L2 Topology And Neighbor Change Counters

Operator question:

• Which devices are reporting topology or neighbor churn?
• How many neighbor rows were inserted, deleted, dropped, or aged out?
• Which switches reported spanning-tree root changes?

Concrete examples:

• LLDP remote-table change traps carry insert, delete, drop, and age-out counts.
• STP root-election traps report topology changes that should be visible per device.

Why it matters:

• Topology-related traps are operationally useful as counters and rates even when Netdata does not mutate topology from traps.
• LLDP table-change traps already carry multiple numeric counters in one notification.
• STP and LLDP churn can indicate loops, cabling changes, switch instability, or monitoring blind spots.

Evidence:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/shanghai-meridian-technologies-co-ltd.yaml:1016 through :1025 defines an LLDP remote-tables-change trap with insert, delete, drop, and age-out counters.
• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/default/dell-inc.yaml:21623 through :21632 defines a Dell STP new-root-election trap.

Use Case 16: Source Investigation And Device Metrics

Scope disposition:

• Source-level investigation is an important operator requirement.
• Built-in receiver health remains job-scoped. Operators investigate individual senders through trap rows and source/enrichment fields.
• Profile-defined metrics provide the opt-in, bounded path for vendor or site semantics that need device-level time series.

Operator question:

• Which source device or relay path is associated with unexpected traps or enrichment decisions?
• Is the listener pipeline healthy in aggregate while one sender's trap rows show a distinct problem?
• Did relay source attribution cause device identity ambiguity?

Concrete examples:

• A single listener receives valid traps from most devices but decode failures from one vendor profile.
• A relay forwards traps and supplies snmpTrapAddress.0; misconfigured trusted relay settings can affect source-device identity.
• INFORM responses can fail for one source even while normal traps continue.

Why it matters:

• Job-level pipeline metrics expose receiver health without creating an always-on series for every sender.
• Source identity is the foundation for all per-device trap metrics.
• Trap rows retain readable source evidence, while profile metrics create only explicitly selected, bounded device-level series.

Evidence:

• src/go/plugin/go.d/collector/snmp_traps/internal/telemetry/job.go collects receiver pipeline, events, severities, errors, and dedup metrics with job_name through an explicit per-job handle.
• TRAP_SOURCE_IP, TRAP_SOURCE_UDP_PEER, and TRAP_ENRICHMENT preserve the source evidence used for log filtering and audit.
• src/go/plugin/go.d/collector/snmp_traps/config.go:26 through :28 defines trusted relay configuration.
• The root Collector.warnCatchAllTrustedRelays composition warning explains that catch-all trusted relays let every peer override source identity via snmpTrapAddress.0.
• internal/jobruntime.Job handles receiver events, including INFORM response failures, and records inform_response_failed through the retained telemetry handle.

Explicit Non-Goals

This use-case inventory does not require the future design to support:

• Dynamic override of journal TRAP_SEVERITY or PRIORITY from varbind values. Severity-aware metrics must not silently change the trap log severity contract.
• A full alarm, incident, or correlation engine. Lightweight trap-derived state is required; OpenNMS-style alarm lifecycle management is not.
• Polling reconciliation, interpolation, or gap filling. Trap-carried numeric values are arrival-time samples, and trap-derived state can be stale.
• Automatic metric generation for every decoded stock trap.
• Arbitrary string, MAC address, IP address, username, free-form text, alarm text, or peer identifier as a metric label by default.
• Trap-driven topology mutation. Topology traps can produce counters or samples, but updating Netdata topology graphs is a separate subsystem.
• Cross-device aggregation inside the collector. Aggregation belongs in query, dashboard, or alerting layers; the collector must preserve per-device identity.
• A general-purpose expression language as a requirement of this use-case set. Filtering is required, but this phase does not prescribe syntax.
• Runtime MIB compilation, a GUI wizard, or profile editing inside the Agent. Operators must be able to provide custom profile files, matching existing trap profile authoring methodology.

Cross-Cutting Practical Constraints

• Metric identity must include source device or an approved equivalent.
• Resource-level metrics must be bounded, stable enough for charts, and operator-understandable.
• Missing varbinds must have explicit behavior.
• Numeric trap samples must distinguish sample value, threshold value, and event count.
• Trap-derived current state must explain stale-state risk.
• Set/clear state must support both separate problem/clear OIDs and same-OID condition varbinds.
• Metric labels must not leak sensitive values or unbounded identifiers.
• Full per-device receiver pipeline monitoring is Phase B of the implementation scope. It MUST reuse the identity and continuous diagnostic hooks needed by profile metric extraction.
• Stock-generated metrics must not create surprising high cardinality by default.
• Custom profiles must not require editing shipped stock files.

Phase 1 Review Questions

External reviewers should answer only use-case coverage questions:

1. Which practical trap-to-metric use cases are missing?
2. Which missing cases are backed by common vendor MIB patterns or established open-source monitoring behavior?
3. Which listed cases are too broad, unsafe, or not useful enough to support?
4. Which examples should be replaced with stronger real-world examples?
5. Which use cases need explicit non-goals to avoid overbuilding?

Reviewers should not propose schema syntax in Phase 1 except when needed to explain a missing use case.

Phase 2 Recommended Design

Summary

Use the existing SNMP trap profile system as the source of truth for trap metric rules. A trap profile should be able to define:

• how a trap is decoded, classified, and rendered;
• which optional metrics can be derived from selected traps;
• how those metrics are filtered, typed, scoped, bounded, and charted.

The job configuration should define runtime policy only:

• listener, authentication, source trust, dedup, rate limit, journal, and OTLP behavior as today;
• whether profile-defined trap metrics are enabled;
• which profile metric rules are explicitly selected.

This is intentionally closer to SNMP polling profiles than Prometheus profiles. Prometheus profiles organize an existing metric stream. Trap profiles must also define extraction, because a trap is an event plus varbinds, not an already emitted metric family.

Operator Workflow

Common operator path:

1. Configure the trap listener as today.
2. Configure source.trusted_relays only when traps pass through relays that must be allowed to set snmpTrapAddress.0.
3. Enable profile-defined trap metrics in the job.
4. List each metric rule to enable by name.
5. Add a custom or override trap profile under /etc/netdata/go.d/snmp.trap-profiles/ only when stock profiles do not provide the desired metric behavior.

Illustrative job configuration:

jobs:
  - name: campus-traps
    listen:
      endpoints:
        - protocol: udp
          address: 0.0.0.0
          port: 162
    versions: [v2c, v3]
    source:
      trusted_relays:
        - 192.0.2.0/24
    profile_metrics:
      enabled: true
      include:
        - IF-MIB::unexpected-link-down-events
        - CISCO-CONFIG-MAN-MIB::cli-config-change-console-events

The responsibilities are:

• profile files define metric semantics;
• jobs define enablement through an explicit rule list;
• source-device identity is mandatory for every device-attributable trap metric.
• profile_metrics: enables profile-defined metric rules.
• The existing job-level trap metrics: key is not retained as a compatibility shim in the clean end state because SNMP traps have not shipped as a public contract.

Safe defaults:

• When profile_metrics is absent, no profile-local metric rules are evaluated.
• profile_metrics.enabled defaults to false.
• Enabling profile metrics requires at least one include entry.
• Missing or profile-disabled rule names fail job creation.
• Device-attributable rules use vnode host scope when enrichment provides an unambiguous vnode; otherwise they use a bounded, hashed fallback source label under the listener job.
• Fixed runtime caps bound selected rules, sources, resources, and chart instances. Overflow drops only the metric instance and increments diagnostics.

Enablement Policy

Profile-defined metrics are disabled by default and use explicit opt-in only. There is no automatic rule selection: profile_metrics.include names every rule selected by the job. A profile rule may set enabled: false; that always makes explicit selection fail validation.

profile_metrics.include entries select metric rule names, not trap names and not profile filenames.

Profile Format Extension

Trap profiles have a first-class optional metrics: section. The section is file-local and validated together with varbinds:, traps:, and the optional profile-local charts: section.

The profile format has one canonical YAML surface. It is the validation, runtime, generated-stock, source-control review, and operator authoring contract. Unknown fields and removed aliases are rejected at profile load.

Canonical shape:

metrics:
  - name: IF-MIB::unexpected-link-down-events
    type: counter
    on_trap: IF-MIB::linkDown
    where:
      - varbind: ifAdminStatus
        equals: up
    identity:
      device: source
      resource:
        class: interface
        key_from_varbind: ifIndex
        max_per_source: 512
    output:
      metric: snmp_trap_if_unexpected_link_down_events
      dimension: events
      chart: unexpected_link_down_events

charts:
  - id: unexpected_link_down_events
    title: Unexpected Link Down Events
    family: Network/Interface/State
    context: snmp.trap.if.unexpected_link_down_events
    units: events/s
    algorithm: incremental
    lifecycle:
      max_instances: 2000
      expire_after_cycles: 60

Canonical rules use these defaults:

• identity.device: source
• identity.resource.key_from_varbind: ifIndex
• identity.resource.max_per_source: 512
• output.metric: deterministic snmp_trap_* name derived from name
• output.dimension: events
• output.chart: deterministic chart ID derived from name
• charts.context: deterministic snmp.trap.* context derived from name
• charts.units: events/s
• charts.algorithm: incremental
• charts.lifecycle: job/profile defaults unless explicitly overridden

Operators can override derived output, chart, identity, or lifecycle fields by writing the canonical field explicitly.

Canonical metric rule fields:

• name: stable metric-rule identity, unique across the effective profile catalogue. Stock rules should use a MIB-qualified stable name; operator rules should use a site-specific prefix to avoid collisions.
• type: one of the supported extraction types.
• identity: source-device scope and optional bounded resource identity.
• output: emitted metric name, dimension name, and referenced chart ID.

Rules MAY omit identity and output when their values can be derived deterministically. Validation rejects derived values that collide with built-in metrics or other enabled profile-local rules.

Type-specific selector fields:

• counter and sample: on_trap is required.
• state with separate problem and clear traps: problem_trap and clear_trap are required, and on_trap is invalid.
• state with same-OID set/clear semantics: on_trap is required together with state.set_when and state.clear_when.
• Trap selectors resolve against traps in the same profile or a lazily routed stock profile. Symbolic names and numeric OIDs must resolve during job Init() before listener startup.

Canonical chart fields:

• id: stable chart ID within the profile file.
• context: chart context, using the snmp.trap.* namespace.
• title, family, units, and algorithm.
• lifecycle for every chart that can create per-source or per-resource instances.

Optional chart fields:

• type: chart type; defaults to the chart-template engine default. State charts should use the clearest supported state/line representation available at implementation time.
• description: operator-facing chart description for generated documentation.

Rule and chart names:

• output.metric names should follow the existing collector metric naming style and must not collide with built-in snmp_trap_* metric names.
• output.metric names must not collide with built-in metric names emitted by the current collector, including event, severity, error, and dedup metric names.
• output.dimension must match the chart-template dimension name that selects the rule's output.metric.
• charts.context values use the chart context namespace and must start with snmp.trap..
• Profile-local chart IDs and contexts must not collide with built-in static chart IDs/contexts for events, severity, errors, or dedup-suppressed metrics.
• output.chart must reference a chart id defined in the same profile file.
• Forward references to charts defined later in that file are allowed.
• output.chart validation happens during profile validation or job Init(). Errors must name the profile file and metric rule. Cross-file chart references are validation errors.

Optional rule fields:

• where: bounded predicates over profile-known varbinds or static trap fields.
• missing: explicit behavior for missing varbinds.
• scale: numeric multiplier/divisor for sampled values, for example scale: { multiplier: 1, divisor: 100 }.
• enabled: make this file's rule available or unavailable for job selection.
• description: author-facing note explaining why the rule exists.

The profile charts: section is a profile-local chart-template description. The loader compiles it into an in-memory charttpl.Spec; unsupported chart-template fields are rejected during profile validation.

Supported Metric Rule Types

The initial design must support these rule types:

• counter: increments a cumulative event counter when the trap and optional predicates match.
• sample: stores the last numeric value extracted from a matching trap for the source/resource identity. Samples are not interpolated and do not imply continuous polling, but active sample series must still be emitted on every periodic Collect() cycle until they expire or are cleared by lifecycle rules.
• state: stores current trap-derived state for a source or source/resource.

The design should not add a separate "multi-value" type. Multiple metric rules may reference the same trap, and chart metadata can group their outputs into one chart. This supports multi-value notifications without making extraction rules harder to validate.

The following examples use the only supported canonical form.

Counter Rule Example

metrics:
  - name: CISCO-CONFIG-MAN-MIB::cli-config-change-console-events
    type: counter
    on_trap: CISCO-CONFIG-MAN-MIB::ccmCLIRunningConfigChanged
    where:
      - varbind: ccmHistoryEventTerminalType
        in: [console, terminal, virtual]
    identity:
      device: source
    output:
      metric: snmp_trap_cisco_cli_config_change_events
      dimension: changes
      chart: cisco_cli_config_changes

charts:
  - id: cisco_cli_config_changes
    title: Cisco CLI Configuration Changes
    family: Configuration/Changes
    context: snmp.trap.cisco.cli_config_changes
    units: events/s
    algorithm: incremental
    lifecycle:
      max_instances: 2000
      expire_after_cycles: 60

Numeric Sample Rule Example

metrics:
  - name: CISCO-PROCESS-MIB::cpu-threshold-current
    type: sample
    on_trap: CISCO-PROCESS-MIB::cpmCPURisingThreshold
    value_from_varbind: cpmCPUTotalMonIntervalValue
    scale: { multiplier: 1, divisor: 1 }
    identity:
      device: source
    output:
      metric: snmp_trap_cisco_cpu_threshold_sample_percent
      dimension: current
      chart: cisco_cpu_threshold

  - name: CISCO-PROCESS-MIB::cpu-threshold-limit
    type: sample
    on_trap: CISCO-PROCESS-MIB::cpmCPURisingThreshold
    value_from_varbind: cpmCPURisingThresholdValue
    scale: { multiplier: 1, divisor: 1 }
    identity:
      device: source
    output:
      metric: snmp_trap_cisco_cpu_threshold_limit_percent
      dimension: threshold
      chart: cisco_cpu_threshold

charts:
  - id: cisco_cpu_threshold
    title: Cisco CPU Threshold Trap Values
    family: System/CPU
    context: snmp.trap.cisco.cpu_threshold
    units: percentage
    algorithm: absolute
    lifecycle:
      max_instances: 2000
      expire_after_cycles: 60

State Rule With Separate Problem/Clear OIDs

metrics:
  - name: IF-MIB::link-down-state
    type: state
    problem_trap: IF-MIB::linkDown
    clear_trap: IF-MIB::linkUp
    identity:
      device: source
      resource:
        class: interface
        key_from_varbind: ifIndex
        max_per_source: 512
    state:
      problem_value: 1
      clear_value: 0
      ttl: 24h
    output:
      metric: snmp_trap_if_link_down_state
      dimension: down
      chart: if_link_down_state

charts:
  - id: if_link_down_state
    title: Trap-Derived Interface Link State
    family: Network/Interface/State
    context: snmp.trap.if.link_down_state
    units: state
    algorithm: absolute
    lifecycle:
      max_instances: 2000
      expire_after_cycles: 60

State Rule With Same-OID Set/Clear Varbind

metrics:
  - name: SNMP-ALARM-MIB::alarm-set-state
    type: state
    on_trap: SNMP-ALARM-MIB::snmpAlarmStatusChange
    identity:
      device: source
      resource:
        class: alarm
        key_from_varbind: snmpAlarmLogId
        max_per_source: 1024
    state:
      set_when:
        varbind: snmpAlarmLogCond
        equals: set
      clear_when:
        varbind: snmpAlarmLogCond
        equals: clear
      problem_value: 1
      clear_value: 0
      ttl: 24h
    output:
      metric: snmp_trap_alarm_set_state
      dimension: set
      chart: snmp_alarm_set_state

charts:
  - id: snmp_alarm_set_state
    title: SNMP Alarm State
    family: Alarms/State
    context: snmp.trap.alarm.state
    units: state
    algorithm: absolute
    lifecycle:
      max_instances: 2000
      expire_after_cycles: 60

Identity And Scoping

Trap metrics must never aggregate all source devices into one listener total unless the metric is explicitly listener-owned and not device-attributable.

Recommended identity model:

• Trap commitment is separate from metric attribution:
  • Accepted traps must be committed to the configured journal and/or OTLP output backend even when source enrichment, vnode attribution, or profile metric attribution is incomplete.
  • Missing SourceVnodeID is not a reason to lose the trap.
  • Metric attribution failures must be visible through continuous receiver-owned diagnostics and TRAP_ENRICHMENT or equivalent log evidence.
• Authoritative output semantics:
  • When both journal and OTLP outputs are enabled, journal commitment is authoritative. OTLP export failures are export errors, not terminal pipeline write failures.
  • When OTLP is the only output backend, OTLP export failures are terminal output write failures.
  • For one accepted trap, pipeline.write_failed increments at most once and only when the authoritative output commit fails. Backend-specific job-level error counters may still record the corresponding failure class.
• Known source device:
  • Use existing trap enrichment to resolve SourceVnodeID.
  • Emit device-attributable trap metrics through V2 host scope for that vnode.
  • Add listener/job labels only as secondary labels where needed for debugging.
• Unknown source device:
  • Emit profile metrics under the receiver's default host scope with bounded source_id and source_kind labels.
  • source_id is derived from the selected trap source identity: trusted snmpTrapAddress.0 when accepted from a trusted relay, otherwise the UDP peer address.
  • synthetic_vnode is a future opt-in mode and must be rejected by the initial schema until it is implemented deliberately.
  • Do not merge unrelated unknown devices into one chart.
  • Do not create unlimited synthetic vnodes by default.
• Ambiguous source device:
  • Commit the trap and preserve source/enrichment evidence in the trap log.
  • Prefer the transport-selected source identity for bounded fallback metrics when it is available and not over cap.
  • Preserve ambiguous source evidence in TRAP_ENRICHMENT.
  • Do not create or migrate vnode-scoped profile metrics from ambiguous vnode enrichment.
  • Ambiguity includes conflicting registry/topology identities for the same trap source, explicit conflict or ambiguous enrichment status, vnode_mismatch or ambiguous_source reasons, rejected candidates, or an original-source address supplied by an untrusted relay.
• Listener-owned diagnostics required by this spec:
  • Keep built-in receiver health at listener/job scope.
  • Emit receiver metrics continuously every Collect() cycle. They must not become sparse just because no trap arrived in a cycle.
  • Keep profile metric extraction and attribution diagnostics job-scoped when profile metrics are enabled.

Built-in static charts for pipeline progress, trap events, severities, processing errors, and dedup suppression remain listener/job-scoped. This keeps receiver health visible for unattributable packets and global listener failures without paying an always-on per-sender series cost.

Operators investigate senders by filtering and grouping trap rows on TRAP_SOURCE_IP and TRAP_SOURCE_UDP_PEER, then inspecting TRAP_ENRICHMENT for relay, ambiguity, and vnode evidence. Per-device vendor semantics are delivered by explicitly selected profile rules.

Fallback profile-metric source identity priority:

1. Trusted snmpTrapAddress.0 only when the UDP peer is a configured trusted relay.
2. UDP peer address.
3. Reverse-DNS or sysName only as display metadata, not as the stable key unless the operator explicitly chooses it.

Required labels for fallback profile metrics:

• job_name

• source_id

• source_kind, for example trusted_trap_address or udp_peer

• source_kind is a closed label set: vnode, listener, trusted_trap_address, udp_peer, entry_source, hostname_or_ip, trap_varbind, topology_ifindex, source, or other. Unknown future enrichment methods map to other.

• Fallback source_id uses a deterministic one-way hash of the canonical source address and job name with the agent's stable local identity as salt; expose only a truncated fixed-length hex value.

• Hashing uses SHA-256, truncates to 16 hexadecimal characters, and canonicalizes addresses without transport ports.

• Hash mode is not a security boundary. Small source-address spaces can be enumerated, and an agent reinstall or stable-local-identity reset changes the salt and therefore changes every hashed source_id.

• The salt source must be a persisted Agent-local identity with restart-stable lifetime. The implementation must document which Agent identity is used; the salt value itself must not be exposed as a metric label.

• Phase A reads /etc/machine-id, then /var/lib/dbus/machine-id, then the hostname, and finally uses a fixed last-resort string only when no stable local identity is available.

The design may later add an explicit opt-in mode for synthetic trap-source vnodes, but that must be an operator decision with caps. It is not the default.

Source identity transitions:

• A source can start unknown and later become known when SNMP polling, registry, or topology enrichment resolves SourceVnodeID.
• New emissions after the transition must use the vnode host scope.
• Existing fallback source-label chart instances are not migrated; they expire through chart lifecycle.
• State entries created under unresolved fallback identity must not be migrated silently to the vnode identity. They are cleared/expired according to the rule's configured TTL and job teardown semantics.
• The transition increments snmp_trap_profile_metrics_source_transitions so chart discontinuity is explainable.
• The source-transition history is diagnostic memory and must be bounded. When it exceeds the fixed source cap, the oldest raw route entries are pruned; losing old transition memory is acceptable, but unbounded growth is not.

Source cap behavior:

• A fixed limit of 2,000 caps tracked non-listener sources per job, including vnode and fallback sources.
• Accepted traps are still committed when source caps are full.
• New profile metric instances for over-cap sources are skipped and counted.
• Inactive sources release cap capacity when their profile metric chart instances expire through chart lifecycle.
• NAT, multi-homed devices, and relays that do not provide trusted original source identity can merge devices under one source address. Operators must use trusted relays or source-device enrichment to avoid that; the collector must not pretend it can split devices it cannot identify.

Identity to host-scope mapping:

• When SourceVnodeID is known, use V2 host scope with ScopeKey=SourceVnodeID and GUID=SourceVnodeID.
• Use the enriched device hostname as host-scope hostname when present.
• All profile metric series include source_id and source_kind, including vnode-scoped series. This keeps one stable chart-template identity across vnode and fallback source modes while V2 host scope remains the primary node attribution.
• When SourceVnodeID is absent, use the bounded fallback source identity under the receiver/default host scope.
• When V2 host scopes are available and a source later resolves to SourceVnodeID, new emissions for that source use vnode host scope.
• Existing fallback source-label chart instances are not migrated; they expire through chart lifecycle. The transition must be visible through a diagnostic counter or log message so chart discontinuity is explainable.

Framework evidence:

• Go V2 host scopes support multiple non-default host scopes from one collector job; series identity includes host scope.
• One chartengine engine is used per host scope, and explicit non-default scopes emit under their metrix.HostScope GUID and metadata.
• Therefore a trap listener job can keep receiver-owned metrics in the default receiver scope while routing device-attributable metrics to per-source vnodes when enrichment supplies them.

Metric Continuity

Netdata does not support sparse receiver metrics. Trap-derived metric state is updated by trap arrival and emitted by the periodic collector cycle.

Required behavior:

• Receiver-owned metrics must be emitted every Collect() cycle regardless of trap arrival in that cycle.
• Counter rules keep cumulative totals per source/resource identity and emit the current total every Collect() cycle while the identity is active.
• State rules keep the last trap-derived state and emit it every Collect() cycle until an explicit clear, TTL expiry, job teardown, or chart lifecycle expiry removes it.
• Sample rules keep the last trap-reported numeric value and emit it every Collect() cycle while the sample is fresh.
• Sample rules must have freshness semantics. The first implementation may use chart lifecycle expiry as the freshness boundary, but it must not emit a stale sample forever without an explicit profile rule or job default.
• A missing varbind with missing: drop does not clear an existing active sample or state value. It only skips the update for the received trap and increments the rule-miss counter.
• A missing varbind with missing: error skips the update and increments the extraction-failure counter.
• New source/resource instances are created only after a matching accepted trap, unless a future approved inventory-seeding design adds pre-created instances.
• Expired source/resource instances are removed. If the same identity appears again after expiry, the next committed trap creates a fresh series; counter charts may show a reset after idle expiry.
• Source/resource cap overflow skips only the new metric instance and increments diagnostics. It must not drop the accepted trap from the trap log/output backend.
• Overflow, eviction, rule-miss, extraction-failure, and attribution-failure diagnostics must themselves be continuous receiver-owned metrics.

Resource Identity

Resource identity is separate from arbitrary labels.

Allowed resource keys:

• Implementation Phase A accepts only integer-like bounded varbind types for identity.resource.key_from_varbind: INTEGER, Integer32, Unsigned32, and Gauge32.
• Table indexes such as ifIndex are valid when capped per source.
• Future support for non-integer peer, neighbor, pool, scope, sensor, MAC, or alarm identifiers requires an approved boundedness design, explicit caps, and documentation before profile authors can use it.

Rejected as default metric labels:

• usernames;
• MAC addresses;
• free-form strings;
• descriptive text;
• peer IPs and source IPs as arbitrary labels;
• unbounded interface names or pool names;
• counter, time-value, string, address, payload-like, and other non-integer resource-key varbinds.

Each resource rule must define:

• class: a stock resource class (interface, peer, neighbor, sensor, alarm, pool, l2_topology, component) or a site-specific lowercase class beginning with site_;
• key_from_varbind or key_from_enrichment;
• optional max_per_source when the fixed default of 512 is too high;
• fixed job-level caps also apply across all sources;
• missing behavior;
• overflow uses the fixed drop-and-count behavior.

Cardinality contract:

Effective runtime caps use the most restrictive applicable limit. A rule cannot create a new source, resource, or chart instance when any applicable job, rule, or chart cap is exhausted.

These caps are collector-enforced before writing to metrix and before the chart-template engine sees the sample. charts.lifecycle.max_instances remains a chartengine defense-in-depth limit because chart template lifecycle caps are best-effort for already-active instances.

The selected-rule limit counts the explicit, post-merge include set. Disabled rules and rules that fail validation cannot be selected.

When the job-level instance cap has one remaining slot and multiple rules would create new instances in the same cycle, the implementation must use a deterministic tie-breaker, for example lexical chart.id then metric rule name. The tie-breaker must be documented and tested.

Initial stock resource classes should be limited to:

• interface
• peer
• neighbor
• sensor
• alarm
• pool
• l2_topology
• component

Operator-defined classes are allowed only with a site-specific prefix. The loader must validate stock classes against the stock list above and operator classes against a documented site-prefix pattern. Stock class validation failures are hard errors. Operator class names must match a documented lowercase identifier pattern and include a site-specific prefix, for example site_foo_sensor.

Overflow drops new source/resource instances beyond the cap and increments a built-in overflow counter. Runtime cap exhaustion does not stop the receiver job or drop accepted traps.

Dropped raw resource keys may be logged at debug level for troubleshooting, but they must not be promoted to labels or durable public artifacts.

Filtering Semantics

The design should support bounded predicates, not a general expression language.

Required operators:

• equals
• in
• exists
• absent
• greater_than
• less_than
• range

Allowed predicate inputs:

• trap name/OID;
• static trap category/severity;
• varbind enum labels;
• booleans;
• small numeric ranges;
• explicit string values only when the referenced varbind is already approved as bounded.

Predicate semantics:

• Multiple predicates are ANDed.
• Each predicate MUST select exactly one string-valued source: varbind or field. Both, neither, and non-string selectors are invalid; use separate predicates for additional AND constraints.
• Each predicate must include at least one condition operator: equals, in, exists, absent, greater_than, less_than, or range.
• A missing varbind in where makes the predicate false and the rule does not match.
• exists: true matches when the referenced varbind is present in the received trap.
• exists: false is equivalent to absent: true.
• absent: true is the only predicate that matches a missing varbind.
• Numeric comparison operators are valid only for numeric varbind types.
• equals and in over enum-backed varbinds match enum labels. equals and in over numeric varbinds match numeric values.
• Numeric predicates evaluate raw decoded numeric values. TimeTicks predicates compare raw hundredths of a second; sample output conversion to seconds is separate.
• range is inclusive on both ends: low <= value <= high.
• Negation is expressed with not: true on a single predicate, not with a general expression language.
• not: true negates the predicate result only after the referenced varbind is known to be present. A missing varbind remains false even when not: true is set.
• {varbind: ifAdminStatus, equals: up, not: true} means "the varbind is present and its value is not up".
• For range, not: true means outside the inclusive range. For in, it means not in the listed set. For greater_than or less_than, it negates that comparison.
• To require absence, use the absent operator instead of not: true with exists.
• Pattern matching is not required for the initial design. Authors should use in over bounded values instead of regular expressions over free-form text.

Missing-varbind behavior must be explicit per rule:

• drop: do not emit the metric and increment a rule-miss counter;
• zero: emit zero only for sample rules that explicitly declare absence means numeric zero;
• unknown_dimension: allowed only for bounded dimensions and capped;
• error: profile validation or runtime error depending on whether the varbind is statically impossible or just absent from a received trap.
• zero is invalid for counter and state rules. State rules must use explicit set/clear predicates or trap pairs.
• The default missing behavior for state rules is drop, which leaves state unchanged.

Sample validation:

• value_from_varbind must reference a profile-known numeric varbind.
• Accepted numeric source types include INTEGER, Integer32, Unsigned32, Gauge32, Counter32, Counter64, and TimeTicks.
• DisplayString, OctetString, MacAddress, IpAddress, OBJECT IDENTIFIER, and free-form textual conventions are rejected for sample rules.
• In the initial design, sample rules must use the absolute chart algorithm. Counter32 and Counter64 trap-carried values may be sampled only as absolute snapshots. Derived counter rates from sporadic trap arrivals are a deferred feature.
• TimeTicks samples are converted to seconds by dividing the raw value by 100 before profile scale is applied.
• Runtime ASN.1 values that do not match the profile numeric type are dropped and counted.
• scale.divisor must be greater than zero.
• The default missing behavior for sample rules is drop.
• Implementation Phase A applies scale before metric emission. Profile authors must set output.metric names, chart titles, and units so the scaled semantics are explicit to operators.

Chart Generation

Profile metric rules should compile into normal go.d V2 chart templates.

Required behavior:

• Every emitted metric must have explicit units and algorithm.
• Counters use cumulative storage and incremental chart algorithm.
• Samples and state metrics use absolute.
• Multiple rules can reference one chart.id when they describe dimensions of the same operational chart.
• Chart instances must include source-device scope or bounded fallback source identity.
• Chart lifecycle must expire stale source/resource instances according to configured caps and TTLs.
• Every generated chart template must pass charttpl.Spec validation.
• context values must use the snmp.trap.* namespace.

Compilation path:

• Profile metric rules and charts are compiled into an in-memory charttpl.Spec.
• ChartTemplateYAML() serves the compiled template; the implementation should not write runtime-generated chart files to satisfy public requests.
• Developer-only debug dumps of the compiled spec are allowed only outside public request paths and must never become the source served by ChartTemplateYAML().
• Static definitions are validated and retained by the shared catalog epoch. Each listener job compiles its explicitly selected rules and generated chart template once during Init(), not on every collection cycle.
• profile_metrics.include validation uses the compiled metric rule catalog.

Chart conflict rules:

• Chart IDs must be unique within a profile file and across the effective profile catalogue; filesystem order never resolves a collision.
• Across the final loaded profile set, two charts may share a context only if title, family, units, algorithm, chart type, and dimension names are compatible.
• Two rules that reference one chart must produce unique dimension names.
• Two rules that reference one chart must have the same label shape. A chart must not mix resource and non-resource rules, and a resource chart must not mix multiple identity.resource.class values.
• Built-in static chart contexts and IDs are reserved; profile-local charts must not reuse events, severity, errors, dedup_suppressed, or profile_metric_diagnostics, nor their effective contexts such as snmp.trap.profile_metric_diagnostics.
• Conflicts must fail profile validation before chartengine planning.

Lifecycle:

• Charts that can create per-source or per-resource instances must declare lifecycle.max_instances and lifecycle.expire_after_cycles.
• expire_after_cycles is measured in the periodic go.d Collect() cycle for the trap listener job. It is not measured by trap receive goroutines. Changing the listener update_every changes the wall-clock lifetime represented by the same cycle count.
• A fixed 50,000-instance job limit is an additional cap across all profile metric charts for the job.
• State TTL sweep runs at the start of each periodic Collect() cycle, not in a background timer and not during per-trap event processing.
• On state TTL expiry, the fixed policy emits the clear value once, removes the state entry, and lets chartengine remove stale chart instances through normal lifecycle planning.
• State rules evaluate where predicates before set/clear logic. If no set/clear predicate matches, the rule increments a rule-miss counter and does not change state.
• State tables must be synchronized; race-free state updates are a required implementation property.
• Multi-endpoint listener jobs can deliver trap metric updates concurrently. The mutex owned by internal/profilemetrics.Runtime serializes state and series mutations within the job.
• State tables are in-memory only. After Agent restart, trap-derived state is unknown until new traps arrive; the implementation must not claim a persisted clear state unless a clear trap was observed after restart.
• A chart's expire_after_cycles must not expire it before state TTL can emit the configured clear value.

This reuses the chart-template engine, but extraction remains a trap-profile responsibility.

Metric Rule Catalog And Lazy Stock Profiles

Profile-defined metric rules must be discoverable before the first matching trap arrives.

Required behavior:

• Job Init() must validate every profile_metrics.include entry against a metric rule catalog before listener startup.
• The metric rule catalog must include operator and stock profile rules.
• The selected rule set must respect the fixed 500-rule limit before the job starts.
• Lazy stock trap decode loading must not delay metric-rule validation until trap arrival.
• The generated stock manifest records metric_rule_names for every profile. During job creation, profile_metrics.include hydrates only the stock files that own the requested rules.
• Every stock manifest entry records a required SHA-256 over the exact decompressed YAML bytes. Lazy hydration verifies and parses the same byte slice so selected rules cannot combine routes from one epoch with profile content from another.
• Operator profiles remain eager. An operator metric rule that references a stock trap by MIB-qualified name hydrates the deterministic candidate-file set routed by that MIB, then requires one exact trap-name match before job creation completes.
• The chosen implementation must preserve the existing lazy decode behavior for jobs that do not enable profile metrics.
• The manifest is the stock metric-rule index. The implementation must not build a second complete in-memory metric catalog or parse the complete stock pack.

Profile Lifecycle

Profiles and metric rules are immutable within a shared catalog epoch while jobs hold leases. After editing operator profiles, restart the Agent or recreate all running snmp_traps jobs. The final lease release unloads the epoch; the next job creation loads profiles and validates profile_metrics.include from scratch.

Runtime Ordering

Trap metric extraction must preserve current collector ordering unless a later design explicitly changes it:

• Decode and render the trap entry.
• Apply dedup admission.
• Do not emit profile metrics for dedup-suppressed traps.
• Write the trap entry to the authoritative output backend.
• Do not emit profile metrics when the write fails.
• Then update profile-defined metrics and built-in static metrics.

Evidence:

• src/go/plugin/go.d/collector/snmp_traps/internal/jobruntime/pipeline.go returns early for dedup-suppressed traps and authoritative write failures, then updates profile, event, and severity metrics only after a successful write.
• src/go/plugin/go.d/collector/snmp_traps/internal/jobruntime/profile_metrics_integration_test.go verifies that profile metrics are not emitted after authoritative write failures or for dedup-suppressed traps.

Composition And Override Semantics

• varbinds:, traps:, metrics:, and charts: are defined and validated as one profile-file bundle; profile files are not field-merged.
• Metric rule names, chart IDs, trap OIDs/names, and metric outputs must be unique across the effective profile catalogue. Filesystem order never decides which definition wins.
• Multiple metric rules may reference the same trap OID.
• An operator profile with the same extensionless filename identity as a stock profile replaces that stock file in full, including decode and metric content.
• An operator profile with a different identity adds complete definitions.
• A metric-only operator profile may reference stock traps without copying the stock profile. enabled: false makes only that file's rule unavailable.
• Partial profile inheritance is unsupported and the extends: key is rejected.

Operator profiles that need to add metrics for traps from several stock files can use one metric-only site profile that references those stock traps by MIB-qualified trap name. If the metric needs varbind validation, resource keys, or sample extraction, the referenced stock trap definition must already define the needed varbind metadata.

Compatibility With Current Job-Level Metrics

The current job-level metrics: list is too limited for the target design.

Approved clean end state:

• SNMP traps have been merged but not released with end-user documentation, so the old job-level trap metrics: list is not treated as a public compatibility contract.
• The implementation should remove or rename the job-level trap metrics: list as needed for the long-term-best configuration model.
• Profile-local metrics: and charts: are the only supported trap metric authoring surface.
• New capabilities belong only in profile-local metrics:.
• Documentation must not teach the obsolete job-level metrics: list.
• Tests that only assert the obsolete per-job trap metric authoring path should be removed or rewritten to validate the profile-local model.

Stock Profile Generation Contract

The stock trap profile generator must not create enabled metrics for every decoded trap.

Required generator behavior:

• Generated decode knowledge remains broad.
• Generated metric rules are absent by default unless curated inputs explicitly request them.
• Generated stock profile YAML must emit canonical metric and chart syntax.
• The generator must validate metric rules against the generated varbinds: and traps: sections before writing YAML.
• Human-curated stock metric rules must be reviewable in source control.
• A regenerated stock profile must preserve curated metric rules or regenerate them from a durable curated source, not lose them silently.
• Stock profile package size and lazy-load memory impact must be checked when adding curated metric rules.

Curated metric rules must have a durable source:

• For generator-owned stock profiles, the preferred durable source is a committed generator input such as curated_metrics.yaml.
• Profile-local YAML blocks maintained directly in source control are acceptable only for profiles that are not regenerated, or when the generator explicitly preserves those blocks in a tested read-modify-write path.

The durable source must record:

• metric rule name;
• referenced trap name/OID;
• referenced varbinds;
• rule type;
• chart ID;
• cardinality evidence.

Metric rules for generated profiles are therefore a curation layer on top of MIB decode generation, not a mechanical "one metric per trap" output.

Validation Requirements

Profile validation must reject:

• unknown top-level profile keys after the profile schema is upgraded; before full profile-wide strictness ships, unknown top-level metrics: and charts: keys must at minimum be rejected instead of silently ignored;
• unknown fields under metrics:;
• unknown fields under charts:;
• every noncanonical alias, map-form predicate, and configurable TTL policy;
• duplicate metric names after merge;
• duplicate output.metric values after merge;
• duplicate or colliding derived output.metric, output.dimension, output.chart, or charts.context values after canonical validation;
• duplicate chart IDs after merge unless handled by the documented replacement rule;
• chart IDs or effective chart contexts that collide with built-in static trap charts;
• metric rules referencing unknown traps;
• metric rules referencing unknown varbinds;
• unsafe labels;
• sample rules without numeric varbind types;
• counters without explicit units/algorithm;
• state rules without set and clear semantics;
• charts with conflicting metadata for the same chart.id;
• where predicates over sensitive varbinds unless the predicate uses an approved bounded enum or boolean representation;
• value_from_varbind targeting known sensitive varbinds;
• output.metric values that collide with built-in metrics emitted by internal/telemetry/job.go;
• output.dimension values that do not match the chart dimension name selecting the rule's output.metric.

Reserved metric name prefixes:

• snmp_trap_events_
• snmp_trap_severity_
• snmp_trap_errors_
• snmp_trap_dedup_
• snmp_trap_pipeline_
• snmp_trap_metric_
• snmp_trap_profile_metrics_

Profile-local rules must not recreate built-in receiver pipeline health. Use profile metrics for vendor or site semantics; built-in receiver metrics cover job-level pipeline progress and processing errors.

The first implementation step that accepts profile-local metrics must:

• add a Metrics field and a Charts field to the profile data model;
• reject or strictly validate unknown metric/chart YAML fields;
• keep metric rules and chart definitions file-local;
• resolve symbolic trap names to canonical OIDs at load time;
• validate every metric rule against the resolved trap's varbind set;
• expose metric validation errors at profile load or job creation time.

The loader must not silently ignore a top-level metrics: or charts: section. Adding profile-local metrics requires updating the profile data model and validation in the same implementation step.

Loader requirements:

• Profile-local metrics: and charts: use strict known-key validation.
• Full top-level strictness should use an audited allowlist of documented profile keys so existing stock profiles are not broken accidentally.
• Validation errors must include the profile filename, the offending key or rule name, and whether the error came from parsing, static validation, or job Init().

Documentation And Skill Updates

The implementation must update every durable surface that currently says trap profiles do not define metrics.

Required updates:

• src/go/plugin/go.d/config/go.d/snmp.trap-profiles/profile-format.md
• .agents/skills/project-snmp-trap-profiles-authoring/SKILL.md
• src/go/plugin/go.d/collector/snmp_traps/config_schema.json
• src/go/plugin/go.d/collector/snmp_traps/metadata.yaml
• generated integration documentation for the SNMP trap listener
• health alert templates under src/health/health.d/snmp_traps.conf when chart contexts, label identity, or vnode scoping changes affect alert matching or alert text

The SNMP trap profile authoring skill must be updated from "profiles do not define metrics" to "trap profiles may define metric rules only through the validated profile-local metrics: and charts: schema described here".

Operator-facing docs, skills, stock generation docs, and contributor docs must present the canonical profile syntax only.

Coverage Matrix

Receiver Pipeline Metrics

The implemented receiver pipeline metrics cover job-level receiver-owned signals such as raw receive rate, accepted/committed rate, drop/error stages, unknown OID/MIB gaps, SNMPv3 USM breakdown, INFORM outcomes, and dedup/throttle suppression. Source-level investigation uses trap logs.

The trap-to-metrics implementation MUST preserve the common contract required by both phases:

• source identity and vnode/fallback attribution for profile-defined metrics;
• accepted trap commitment before metric attribution;
• continuous extraction diagnostics for attribution failures, rule misses, extraction failures, cap overflows, and source route transitions.

Receiver/pipeline metrics MUST NOT silently drop accepted traps when enrichment, profile matching, source attribution, or metric extraction fails. Those failures produce diagnostics and/or log evidence; they are not reasons to discard the trap.

Receiver/pipeline metrics MUST emit continuously at job scope. Profile-defined source/resource instances MUST remain bounded by explicit caps and lifecycle rules; receiver-level totals MUST remain available for unattributable errors and global listener state.

Required Test Coverage

The implementation must retain tests for:

• profile-local metrics: parsing and strict validation;
• profile-local charts: parsing and strict validation;
• canonical list-form predicates and nested identity/output/state syntax;
• rejection of every removed alias and configurable policy field;
• derived metric name, dimension, chart ID, and context collision rejection;
• unknown top-level key rejection or targeted rejection for metrics: and charts: before runtime support is complete;
• rejection of the removed extends: key and duplicate-name detection across independent profiles;
• chart merge, chart context conflict, duplicate dimension, and charttpl.Spec validation;
• built-in static chart ID/context collision rejection;
• metric rule output.chart references to same-file chart IDs;
• rejection of cross-file chart references;
• metric rule catalog construction for lazy stock profiles and include validation during job Init();
• fixed selected-rule limit after catalogue selection;
• multiple metrics referencing the same trap OID;
• symbolic trap name and numeric OID resolution order;
• missing-varbind behavior;
• bounded filter predicates;
• exists and absent predicate semantics;
• numeric comparison predicates and invalid numeric predicates on non-numeric varbinds;
• range predicate inclusive bounds;
• not: true semantics, including missing-varbind behavior and absence via absent;
• numeric sample extraction, scaling, and non-numeric rejection;
• TimeTicks conversion to seconds;
• rejection of incremental algorithm for initial sample rules;
• separate-OID state pairs;
• same-OID set/clear state;
• state TTL sweep and race-free state updates;
• state reset semantics after restart;
• concurrent state updates from multiple trap-processing goroutines hitting the same rule and resource key;
• per-source identity with known SourceVnodeID;
• V2 host-scope emission for known SourceVnodeID;
• fallback source-label emission when SourceVnodeID is absent;
• transition from unresolved fallback identity to known SourceVnodeID;
• fixed hashed fallback source-label and overflow behavior;
• accepted trap commitment when profile metric attribution fails or cap overflow skips new metric instances;
• receiver pipeline counters for received, decoded, accepted, committed, dedup-suppressed, dropped, and write-failed traps;
• continuous emission of receiver counters, profile counters, state values, and fresh sample values across Collect() cycles with no new traps;
• hash privacy stability across restarts and absence of raw source label leakage in hash mode;
• ambiguous source handling;
• resource caps and overflow counters;
• rule-miss counters for missing predicates and extraction failures;
• sensitive/high-cardinality label rejection;
• sensitive/high-cardinality predicate and value_from_varbind rejection;
• built-in metric-name collision rejection;
• generated chart templates and chartengine planning;
• collector-enforced caps before charttpl best-effort lifecycle behavior;
• V2 host-scope routing for known source devices;
• config schema updates for profile_metrics;
• generated metadata/integration documentation updates;
• health alert compatibility when chart identity or labels change;
• profile generator preservation of curated metric rules;
• shared catalog epoch behavior when profiles change between the final lease release and next acquisition;
• job restart behavior for profile_metrics configuration changes;
• dedup/write-failure runtime ordering;
• rejection or removal of the obsolete job-level trap metrics: authoring path;
• per-cycle overhead benchmark near fixed caps, covering rule evaluation, hashing, state updates, resource cap checks, and TTL sweep. The benchmark must report time and allocations and define an implementation-specific regression budget before release.

Historical Phase 2 Review Questions

External reviewers should answer design questions:

1. Does the design cover every use case without adding a full alarm engine or topology mutation system?
2. Is profile-local metrics: the simplest model consistent with SNMP profile methodology?
3. Are the identity rules strong enough to satisfy per-device metrics without dangerous cardinality?
4. Are any schema concepts unnecessary, ambiguous, or weaker than existing Netdata/SNMP profile patterns?
5. What unwanted side effects could this design create in loader behavior, generated stock profiles, chart templates, docs, or backward compatibility?