docs/npm/snmp-traps/troubleshooting.md
Use this page when traps do not appear, appear with unexpected fields, or stop reaching the backend you rely on. Jump straight to the section that matches what you see:
received flatlinedunknown categoryStart with receiver metrics, then inspect rows, configuration, and packet delivery. Deep packet or device debugging comes last, because the receiver pipeline usually tells you where the packet stopped.
Open Netdata Metrics or Charts and search for SNMP trap receiver pipeline or snmp.trap.pipeline. Select the affected listener job, then walk the funnel from the top — the first dimension that stops rising is where to look:
flowchart TD
S(["snmp.trap.pipeline
for the affected job"]) --> Q1{"received rising?"}
Q1 -->|no| K["ipv4.udperrors RcvbufErrors,
listener bound, port, firewall, device"]
Q1 -->|yes| Q2{"decoded rising?"}
Q2 -->|no| V["pre-decode rejects:
allowlist, version, auth, malformed"]
Q2 -->|yes| Q3{"accepted rising?"}
Q3 -->|no| P["post-decode policy:
community, engine-id, rate-limit"]
Q3 -->|yes| Q4{"committed rising?"}
Q4 -->|no| B["backend: dedup_suppressed,
journal/OTLP health, write_failed"]
Q4 -->|yes| OK["pipeline healthy —
inspect rows and enrichment"]
The table below details each branch:
| If this is happening | First place to look | Likely area |
|---|---|---|
received is flat or flatlined under load | ipv4.udperrors (RcvbufErrors) first, then listener job, bind address, port, device destination, network path | No packets are entering the trap handler. |
received rises but decoded is flat | snmp.trap.errors, decode-error rows, SNMP version, community, SNMPv3 settings | Packets arrived but did not decode as allowed traps. |
decoded rises but accepted is flat | dropped_allowlist, rate_limited, unknown_engine_id, source policy, SNMP version/community, SNMPv3 engine ID, rate-limit settings | Packets decoded but did not become accepted trap entries. |
accepted rises but committed is flat | dedup_suppressed, write_failed, journal_write_failed, otlp_export_failed, deduplication settings, output backend health | Entries were accepted but not written or exported as normal rows. |
dedup_suppressed rises | Deduplication summary rows and dedup configuration | Repeated traps are intentionally summarized. |
write_failed rises | Direct journal or OTLP backend health | The configured output backend is failing. |
Use the SNMP trap processing errors chart, snmp.trap.errors, next. For the full dimension list and what each one means, see Metrics.
Some dimensions are diagnostic, not pipeline blockers. inform_response_failed means Netdata could not send an INFORM acknowledgement, but the trap can still continue through the receiver. binary_encoded means journal fields were encoded safely for storage; it does not by itself mean packet loss. listener_read_failed means the listener socket returned a read error.
Every per-symptom check below reads the affected job's journal. The canonical command is the same each time — only the filter changes:
sudo journalctl --directory=/var/log/netdata/traps/<job>/$(tr -d '-' < /etc/machine-id) \
--since "2 hours ago" --no-pager
Replace <job> with your listener job name (examples below use edge-traps). To narrow to a row type or field, add a FIELD=value match before the flags — for example TRAP_REPORT_TYPE=decode_error — and optionally --output=json-pretty --output-fields=... to project specific fields. The sections below show only the filter to add.
You expected trap rows and see none.
snmp:traps Function; OTLP-only jobs do not — query the OTLP receiver instead.received is flat, go to Listener or job not active and UDP packets are not reaching Netdata. If received rises, check snmp.trap.errors before changing device configuration.TRAP_REPORT_TYPE=trap to see only normal traps, or TRAP_REPORT_TYPE=decode_error (with the decode-error field projection below) to see packets that arrived but could not be decoded:... TRAP_REPORT_TYPE=decode_error --since "2 hours ago" --output=json-pretty \
--output-fields=__REALTIME_TIMESTAMP,MESSAGE,TRAP_JOB,TRAP_DECODE_ERROR_KIND,TRAP_DECODE_ERROR,TRAP_VERSION,TRAP_SOURCE_IP,TRAP_SOURCE_UDP_PEER,TRAP_PACKET_SIZE,TRAP_PACKET_SHA256,TRAP_LISTENER,TRAP_ENGINE_ID,TRAP_JSON --no-pager
See Journal and Querying and Field Reference.
SNMP trap collection is explicit. The stock file is a commented template, so a listener is not active until an enabled jobs: entry exists in:
/etc/netdata/go.d/snmp_traps.conf
Depending on installation type, paths may be prefixed with /opt/netdata.
Check:
jobs:.listen.endpoints contains the local address and UDP port the devices send to.journal.enabled: true, otlp.enabled: true, or both.Conservative checks:
sudo systemctl status netdata --no-pager
sudo journalctl -u netdata \
--since "30 minutes ago" \
--no-pager
If the listener port is below 1024, such as UDP 162, the process needs permission to bind it. Netdata packages grant the needed capability to go.d.plugin. For listener configuration details, see Configuration.
This is the likely path when received is flat for the job.
Check ipv4.udperrors first. The collector's received counter counts packets after the kernel UDP receive buffer, so datagrams the kernel drops never appear in the trap pipeline — traps can flatline while the device is still sending. Netdata's system-level ipv4.udperrors chart records these on its RcvbufErrors dimension (the 1m_ipv4_udp_receive_buffer_errors alert watches it). If RcvbufErrors is climbing, the kernel buffer is your bottleneck, not the network; tune it as described in Sizing and Capacity.
Then check the receiving side:
Check local UDP listeners:
sudo ss -ulnp | grep ':162'
If the job listens on a non-privileged test port, replace 162 with that port.
Then check the sending side:
Do not start by changing communities or SNMPv3 keys when received is flat. Credential mismatches usually require packets to reach the receiver first.
allowlist.source_cidrs is checked against the UDP peer before the packet is parsed. If a relay forwards traps, the allowlist must include the relay UDP peer, not only the original device address carried inside the trap.
Evidence:
snmp.trap.errors dimension dropped_allowlistsnmp.trap.pipeline dimension droppedallowlist.source_cidrsCommon causes:
Fix the allowlist with the narrowest CIDRs that match the actual UDP peers:
allowlist:
source_cidrs:
- 192.0.2.0/24
- 198.51.100.0/24
For source controls, see Configuration.
For SNMPv1 and SNMPv2c, check the configured versions and community allowlist without exposing the real community value.
Evidence:
received rises while decoded is flat when Netdata can identify a disallowed version before full decode.decoded rises while accepted is flat when a decoded v1/v2c trap uses a version or community that the job does not allow.snmp.trap.errors has dropped_allowlist.TRAP_VERSION when Netdata can safely read it from malformed or otherwise undecodable packets.Check:
versions includes the version the device sends, such as v2c.communities uses secret references or placeholders, not inline real values.communities: [] accepts all SNMPv1/v2c communities and should be used only when that is intentional.Safe example:
versions:
- v2c
communities:
- "${file:/run/secrets/snmp-trap-community}"
Do not paste real community strings into tickets, docs, shell history, or examples.
Common cue: v3 is silent but v2c from the same device works. When v2c traps arrive and v3 traps do not, the listener is reachable, so the problem is almost always SNMPv3-specific: a USM credential mismatch (username, auth/privacy protocol, or key) or an engine-ID mismatch. Read the receiver's own signals first — the usm_failures and unknown_engine_id error dimensions on snmp.trap.errors, and the decode-error rows (TRAP_REPORT_TYPE=decode_error) with TRAP_DECODE_ERROR_KIND, TRAP_DECODE_ERROR, and TRAP_ENGINE_ID — rather than reaching for external SNMP tools.
After a device reboot, v3 can reject for a while. A reboot resets the sender's engine boots/time. If the receiver still has the pre-reboot engine time cached for that engine ID, time-window checks can reject otherwise-valid messages until the new engine state is learned, showing up as usm_failures. If v3 was working and started failing right after a known device restart, correlate with a coldStart/warmStart trap from the same source and allow the engine state to resynchronize before assuming a credential change.
SNMPv3 failures usually show up as one of these error dimensions:
auth_failuresusm_failuresunknown_engine_iddecode_failedQuery decode errors for the affected job with the canonical command:
... TRAP_REPORT_TYPE=decode_error --since "2 hours ago" --output=json-pretty \
--output-fields=__REALTIME_TIMESTAMP,MESSAGE,TRAP_DECODE_ERROR_KIND,TRAP_VERSION,TRAP_SOURCE_IP,TRAP_SOURCE_UDP_PEER,TRAP_ENGINE_ID,TRAP_JSON --no-pager
Check:
versions includes v3.usm_users[].username matches the device.auth_proto and priv_proto match the device.auth_key and priv_key resolve from secret references and are not pasted inline.engine_id_whitelist.dynamic_engine_id_discovery: true and an empty engine_id_whitelist.dynamic_engine_id_max_pairs, has not rejected new (engineID, username) pairs.TRAP_ENGINE_ID is not a password, but it is inventory data. Treat it as sensitive operational context.
For SNMPv3 configuration, see Configuration.
Malformed packets reached the listener but could not become normal trap rows.
Evidence:
snmp.trap.errors dimensions malformed_pdu or decode_failedTRAP_REPORT_TYPE=decode_errorTRAP_DECODE_ERROR_KINDTRAP_PACKET_SIZETRAP_PACKET_SHA256TRAP_LISTENERUse the packet hash (TRAP_PACKET_SHA256) to group repeated bad packets without storing raw bytes. With the canonical command:
... TRAP_REPORT_TYPE=decode_error --since "2 hours ago" --output=json-pretty \
--output-fields=__REALTIME_TIMESTAMP,TRAP_DECODE_ERROR_KIND,TRAP_PACKET_SIZE,TRAP_PACKET_SHA256,TRAP_LISTENER,TRAP_SOURCE_IP,TRAP_SOURCE_UDP_PEER,TRAP_JSON --no-pager
Check:
Raw packet captures can contain communities, engine IDs, hostnames, interface names, and device payload. Use them only under your local security process and do not share raw captures in public issues.
Unknown OIDs are coverage signals, not always ingestion failures. A trap can be accepted and committed while still using TRAP_CATEGORY=unknown.
Evidence:
snmp.trap.errors dimension unknown_oidTRAP_OID but missing or unexpected TRAP_NAMETRAP_CATEGORY=unknownsnmp.trap.errors dimensions profile_load_failed or template_unresolvedCheck profile locations:
/etc/netdata/go.d/snmp.trap-profiles/
/usr/lib/netdata/conf.d/go.d/snmp.trap-profiles/default/
Depending on installation type, paths may be prefixed with /opt/netdata.
Common causes:
For custom profile behavior and safe profile changes, see Trap Profiles and Validation and Data Quality.
Thousands of identical traps, or fewer rows than packets sent: rate limiting and deduplication intentionally change what appears as rows.
snmp.trap.errors rate_limited, snmp.trap.pipeline dropped, and job rate_limit config.snmp.trap.pipeline dedup_suppressed, snmp.trap.dedup_suppressed suppressed, rows with TRAP_REPORT_TYPE=deduplication_summary, and fields TRAP_SUPPRESSED_COUNT, TRAP_SUPPRESSED_FINGERPRINTS, TRAP_REPORT_PERIOD_SEC.Query dedup summaries with the canonical command:
... TRAP_REPORT_TYPE=deduplication_summary --since "2 hours ago" --output=json-pretty \
--output-fields=__REALTIME_TIMESTAMP,MESSAGE,TRAP_JOB,TRAP_SUPPRESSED_COUNT,TRAP_SUPPRESSED_FINGERPRINTS,TRAP_REPORT_PERIOD_SEC,TRAP_JSON --no-pager
Check:
rate_limit.enabledrate_limit.mode, especially drop versus samplededup.enableddedup.window_secdedup.key_varbindsIf operators expect every repeated PDU to appear as an individual row, deduplication is the first setting to check. Dedup-suppressed traps do not update profile-defined metrics.
No local rows for a direct-journal job, or write_failed rising. Committed rows are written under the per-job root /var/log/netdata/traps/<job>/ (or ${NETDATA_LOG_DIR}/traps/<job>/); journalctl --directory reads the machine-id child of that root.
Evidence: snmp.trap.pipeline write_failed, snmp.trap.errors journal_write_failed, a missing/unreadable journal directory, or no local trap log source for a direct-journal job.
Check the job output mode first:
journal.enabled: true means local files should exist.journal.enabled: false with otlp.enabled: true means OTLP-only; no local source is expected.Conservative checks — list the directory (sudo ls -lh /var/log/netdata/traps/edge-traps), read it with the canonical command, and confirm disk space (df -h /var/log/netdata).
Common causes:
For journal querying and local source behavior, see Journal and Querying.
OTLP export is optional and disabled by default. When enabled, failures show as:
snmp.trap.errors dimension otlp_export_failedsnmp.trap.pipeline dimension write_failed when OTLP is the failing authoritative write pathIn journal+OTLP jobs, secondary OTLP export failures raise otlp_export_failed but do not remove rows already accepted by the journal backend. In OTLP-only jobs, terminal OTLP failures raise write_failed with otlp_export_failed.
Check:
otlp.enabled and otlp.endpointhttps:// for TLS, http:// or bare host:port for plaintext gRPCotlp.headersrequest_timeout, flush_interval, batch_size, queue_capacity) — see ConfigurationSafe example:
otlp:
enabled: true
endpoint: "https://otel-collector.example.net:4317"
headers:
authorization: "${file:/run/secrets/snmp-trap-otlp-authorization}"
If the job is OTLP-only, do not expect local trap rows in the snmp:traps Function. Query the downstream system.
Use row fields before changing enrichment configuration.
Start with TRAP_SOURCE_IP, TRAP_SOURCE_UDP_PEER, _HOSTNAME, TRAP_DEVICE_VENDOR, TRAP_INTERFACE, TRAP_NEIGHBORS, TRAP_REVERSE_DNS, TRAP_ENRICHMENT, and ND_NIDL_NODE.
Compare selected source and UDP peer with the canonical command:
... TRAP_SOURCE_IP=192.0.2.10 --since "2 hours ago" --output=json-pretty \
--output-fields=__REALTIME_TIMESTAMP,MESSAGE,TRAP_SOURCE_IP,TRAP_SOURCE_UDP_PEER,_HOSTNAME,TRAP_DEVICE_VENDOR,TRAP_INTERFACE,TRAP_NEIGHBORS,TRAP_REVERSE_DNS,TRAP_ENRICHMENT --no-pager
Check:
TRAP_SOURCE_IP and TRAP_SOURCE_UDP_PEER.TRAP_SOURCE_UDP_PEER.source.trusted_relays may make snmpTrapAddress.0 become the selected source.TRAP_REVERSE_DNS is annotation only. It is not authoritative identity.TRAP_ENRICHMENT explains selected, rejected, missing, ambiguous, or skipped enrichment decisions.TRAP_ENRICHMENT for ambiguous, rejected, or changing source evidence.Keep source.trusted_relays narrow. A broad trusted-relay range lets senders on that path influence source attribution.
For field meanings, see Field Reference. For source validation workflow, see Validation and Data Quality.
Retention applies only when journal.enabled is true.
Check:
retention.max_sizeretention.max_durationretention.rotation_sizeretention.rotation_durationjournal_write_failedConservative checks — confirm disk space (df -h /var/log/netdata), measure the job's footprint (sudo du -sh /var/log/netdata/traps/edge-traps), and read the window with the canonical command.
Common causes:
Choose retention based on how long operators need local forensic access. If an external backend is the system of record, validate that backend before reducing local retention.
192.0.2.10, 198.51.100.20, and 203.0.113.5.TRAP_JSON, TRAP_ENRICHMENT, TRAP_VAR_*, TRAP_ENGINE_ID, interface names, neighbor names, and hostnames before sharing output.