docs/npm/snmp-traps/validation-and-data-quality.md
Use this page before relying on SNMP trap logs during incidents. The goal is to prove that trap collection is correct, secure, and useful: packets arrive, source identity is trustworthy, profiles add useful meaning, storm controls behave as intended, and the selected output backend keeps the events you need.
Validation is not only "did a row appear?" A quiet receiver can be healthy, misconfigured, blocked by network policy, dropping unauthorized packets, or waiting for devices that have not been configured to send traps. Validate trap rows and receiver metrics together.
Work through these checks in order during first deployment. Use Quick Start to send a known test trap, and use Journal and Querying when you need local journalctl or Cloud Logs verification commands.
| Check | Evidence to inspect | What good looks like |
|---|---|---|
| First receipt | TRAP_REPORT_TYPE, TRAP_JOB, TRAP_OID, TRAP_NAME, TRAP_SOURCE_IP, snmp.trap.pipeline | A known test trap appears as TRAP_REPORT_TYPE=trap, and received, decoded, accepted, and committed move for the listener job. |
| Source identity | TRAP_SOURCE_IP, TRAP_SOURCE_UDP_PEER, TRAP_ENRICHMENT.source | Direct senders usually have matching source and UDP peer. Relayed traps show why Netdata selected the reported source. OTLP-only jobs use equivalent OTLP attributes instead of journal field names. |
| Credentials and version | TRAP_VERSION, TRAP_REPORT_TYPE=decode_error, TRAP_DECODE_ERROR_KIND, snmp.trap.errors | Expected SNMP versions are accepted. Authentication, USM, unknown engine ID, and malformed packet errors stay at expected levels. |
| Allowed sources and communities | TRAP_SOURCE_UDP_PEER, snmp.trap.errors dropped_allowlist, job allowlist.source_cidrs, job communities | Only intended UDP peers are admitted. Only intended SNMPv1/v2c communities are accepted once the packet is parsed. |
| Trusted relays | TRAP_SOURCE_IP, TRAP_SOURCE_UDP_PEER, TRAP_ENRICHMENT.source.method, TRAP_ENRICHMENT.source.snmp_trap_address, TRAP_ENRICHMENT.source.trusted_relay | Direct senders use udp_peer. Only configured relay peers can make snmpTrapAddress.0 become the selected source. |
| Profile coverage | TRAP_NAME, MESSAGE, TRAP_OID, TRAP_CATEGORY, TRAP_SEVERITY, snmp.trap.errors unknown_oid | Common device traps resolve to names, rendered messages, categories, severities, and useful varbind labels. Unknown OIDs are tracked as coverage gaps. |
| Profile load health | snmp.trap.errors profile_load_failed, template_unresolved | Loaded profile YAML files parse cleanly, and profile-rendered templates resolve the varbinds and fields they reference. |
| Severity and category policy | TRAP_CATEGORY, TRAP_SEVERITY, TRAP_TAG_*, overrides | The resulting urgency and category match local operations policy for the trap OID. |
| Enrichment quality | TRAP_ENRICHMENT, _HOSTNAME, TRAP_DEVICE_VENDOR, TRAP_INTERFACE, TRAP_NEIGHBORS, TRAP_REVERSE_DNS | Enrichment fields appear when local context exists, and the audit explains skipped, missing, or ambiguous lookups. |
| Varbind quality | TRAP_VAR_*, TRAP_JSON, snmp.trap.errors template_unresolved, binary_encoded | Important event payload is queryable through indexed fields, with structured payload and audit details available in TRAP_JSON; binary-encoded journal fields are counted and need review. |
| Deduplication | TRAP_REPORT_TYPE=deduplication_summary, TRAP_SUPPRESSED_COUNT, TRAP_SUPPRESSED_FINGERPRINTS, snmp.trap.dedup_suppressed | Repeated traps are summarized only when deduplication is intentionally enabled and the fingerprint matches operational expectations. |
| Rate limiting | snmp.trap.errors rate_limited, job rate_limit.mode | Over-limit traffic is counted by rate_limited. In drop mode the packet is discarded; in sample mode it is counted and still enters the pipeline. |
| Journal health | snmp.trap.pipeline committed, write_failed, snmp.trap.errors journal_write_failed, direct journal path | Healthy direct-journal jobs commit rows and do not report write failures. Local journalctl validation requires journalctl on the host where you query the files. |
| OTLP health | snmp.trap.errors otlp_export_failed, downstream receiver records, OTLP resource attributes | OTLP export succeeds, records have service.name=netdata-snmptrap and the expected service.instance.id, and downstream records contain the expected trap attributes. |
| Retention | Job retention settings and direct journal files | Local retention keeps enough trap history for investigation without unexpectedly evicting needed rows. |
| Downstream SIEM fields | Ingested field names or OTLP attributes | SIEM rules use the fields that actually arrive, especially report type, source, OID/name, category, severity, and payload fields. |
Always inspect TRAP_REPORT_TYPE first:
trap means Netdata decoded and accepted an SNMP Trap or INFORM and wrote it to the configured output.decode_error means a packet reached the decode-error path but did not become a normal trap row. Inspect TRAP_DECODE_ERROR_KIND, TRAP_DECODE_ERROR, TRAP_VERSION, TRAP_SOURCE_IP, TRAP_SOURCE_UDP_PEER, TRAP_SOURCE_UDP_PORT, TRAP_PACKET_SIZE, TRAP_PACKET_SHA256, TRAP_LISTENER, TRAP_ENGINE_ID, and TRAP_JSON.deduplication_summary means repeated matching traps were suppressed during a deduplication period. Inspect TRAP_SUPPRESSED_COUNT, TRAP_SUPPRESSED_FINGERPRINTS, TRAP_REPORT_PERIOD_SEC, and TRAP_JSON.Drops that happen before a row is written may be visible only in receiver metrics. Use snmp.trap.pipeline, snmp.trap.errors, and snmp.trap.dedup_suppressed together with the log rows and their source fields.
Source identity controls attribution, filtering, enrichment, and incident routing. Validate it before using trap data for paging or audit workflows.
Compare TRAP_SOURCE_IP with TRAP_SOURCE_UDP_PEER (equal for direct senders, different for a trusted-relay override), then inspect the source object inside TRAP_ENRICHMENT to confirm selected, method, trusted_relay, and any rejected_candidates. For the full source-identity model and the step-by-step trust checklist, see Enrichment. Keep source.trusted_relays narrow: a broad range lets senders on that path influence attribution through snmpTrapAddress.0.
Treat TRAP_REVERSE_DNS as a label only; keep trust decisions based on TRAP_SOURCE_IP, TRAP_SOURCE_UDP_PEER, and TRAP_ENRICHMENT.source.
For OTLP-only jobs, validate the same source relationship with OTLP attributes: snmp.source.ip is the selected source and network.peer.address is the UDP peer. Use the Field Reference for the journal-to-OTLP mapping.
The listener should accept only the SNMP versions and credentials that devices are expected to use.
TRAP_VERSION, TRAP_ENGINE_ID, and error dimensions such as auth_failures, usm_failures, and unknown_engine_id.TRAP_DECODE_ERROR_KIND=auth_failures, usm_failures, or unknown_engine_id usually point to sender or listener credential mismatch, engine ID policy, or unauthorized senders.malformed_pdu, decode_failed, auth_failures, usm_failures, and unknown_engine_id when triaging.Profile coverage determines whether rows are useful without manual OID lookup.
For normal trap rows, inspect:
TRAP_OID: the numeric trap identifier.TRAP_NAME: the resolved profile or MIB-qualified name, when known.MESSAGE: the rendered trap message shown in log views.TRAP_CATEGORY: the operational category.TRAP_SEVERITY: the operational severity.TRAP_VAR_*: indexed event varbinds for filtering and rule logic.TRAP_JSON: structured payload and audit details.An unknown OID is a coverage signal, not an ingestion failure. Netdata can still accept and store the trap with TRAP_OID and TRAP_CATEGORY=unknown. Use snmp.trap.errors unknown_oid, missing TRAP_NAME, and repeated important raw OIDs to decide whether to add or adjust trap profiles or local overrides.
Profile load and template health are separate from OID coverage. Use snmp.trap.errors profile_load_failed for profile files that cannot be loaded, and template_unresolved for rendered text that references missing varbinds or fields.
Validate severity and category as policy, not only parsing. A trap can be decoded correctly while still needing a local override because your organization treats that OID differently. Keep category and severity inside the documented closed values so queries, charts, and downstream rules remain stable.
Enrichment should help operators understand the row faster. It should not hide the original evidence.
Use concrete fields for normal filtering:
_HOSTNAMETRAP_DEVICE_VENDORTRAP_INTERFACETRAP_NEIGHBORSTRAP_REVERSE_DNSND_NIDL_NODEThese fields are conditional. They appear only when Netdata has enough local context to populate them, such as a matched device, interface, neighbor, reverse-DNS result, or topology node. If a field is absent, use TRAP_ENRICHMENT.registry, TRAP_ENRICHMENT.topology, TRAP_ENRICHMENT.interface, and TRAP_ENRICHMENT.neighbors to see whether the lookup was missing, ambiguous, skipped, pending, or in conflict.
Use TRAP_ENRICHMENT when a field is missing or surprising. It records source selection, local device lookup, interface lookup, neighbor lookup, reverse-DNS state, and applied fields. Common useful statuses include matched, no_match, ambiguous, skipped, pending, and conflict.
Treat enrichment as context. The selected source fields and the original trap OID remain the primary evidence for what Netdata received.
Rate limiting and deduplication change what you see in logs.
snmp.trap.errors rate_limited. With rate_limit.mode: drop, over-limit packets are discarded; with rate_limit.mode: sample, they are counted and still continue through the receiver.TRAP_REPORT_TYPE=deduplication_summary rows. Check dedup.window_sec, dedup.cache_max_entries, and dedup.key_varbinds in Configuration when summaries do not match expectations.TRAP_SUPPRESSED_COUNT, TRAP_SUPPRESSED_FINGERPRINTS, TRAP_REPORT_PERIOD_SEC, and TRAP_JSON to understand what was suppressed.Dedup-suppressed traps are counted in periodic summary entries instead of stored as individual rows. If an operator expects every repeated PDU to be visible, deduplication is the first setting to check.
For direct-journal jobs:
snmp:traps Function.snmp:traps Function behavior, Cloud requirements, and local journalctl --directory commands.snmp.trap.pipeline committed moves with accepted traps.snmp.trap.pipeline write_failed and snmp.trap.errors journal_write_failed.For OTLP export:
snmp.trap.errors otlp_export_failed stays clear.service.name=netdata-snmptrap and service.instance.id=<job>. See Forwarding to SIEM for the OTLP export checks.If both direct journal and OTLP are enabled, validate both paths. If a job is OTLP-only, it will not create local journal files and will not appear as a local job source in the snmp:traps Function.
Before building SIEM rules, decide whether the SIEM receives direct-journal fields or OTLP attributes. The names differ.
For direct-journal ingestion, validate that the SIEM preserves at least:
TRAP_REPORT_TYPETRAP_JOBTRAP_SOURCE_IPTRAP_SOURCE_UDP_PEERTRAP_OIDTRAP_NAMETRAP_CATEGORYTRAP_SEVERITYTRAP_VERSIONTRAP_VAR_*TRAP_JSONTRAP_REPORT_TYPE=decode_errorTRAP_REPORT_TYPE=deduplication_summaryFor OTLP ingestion, use the OTLP mapping in Field Reference. Build rules against report type, selected source, UDP peer, trap OID, trap name, category, severity, version, varbind payload, decode-error details, and dedup summary attributes.
Avoid making TRAP_JSON or TRAP_ENRICHMENT the primary SIEM grouping key. They are audit payloads and can be large or high-cardinality. Prefer specific stable fields, then inspect the JSON payload when the indexed fields do not answer the question.
Silence has several meanings:
allowlist.source_cidrs rejects the UDP peer.Do not treat a quiet receiver as proof of device health. Send a known test trap, check snmp.trap.pipeline, inspect snmp.trap.errors, and verify the configured output backend.
Trap data can include sensitive network and device context. Validate useful data, but minimize what you copy, forward, or expose. Treat communities, SNMPv3 keys, and OTLP headers as credentials (use secret references), and review TRAP_JSON, TRAP_VAR_*, and TRAP_ENRICHMENT before forwarding because device-provided varbinds and enrichment can carry usernames, interface descriptions, MACs, asset tags, locations, or public addresses. For the complete per-field sensitivity guidance, see Field Reference.
Use RFC 5737 example IPs such as 192.0.2.10, 198.51.100.20, and 203.0.113.5 in examples. Use placeholders for secrets, device names, organization names, and private endpoints.
journalctl and Cloud Logs query workflows.