Documentation/networking/devlink/devlink-trap.rst
.. SPDX-License-Identifier: GPL-2.0
Devices capable of offloading the kernel's datapath and perform functions such as bridging and routing must also be able to send specific packets to the kernel (i.e., the CPU) for processing.
For example, a device acting as a multicast-aware bridge must be able to send IGMP membership reports to the kernel for processing by the bridge module. Without processing such packets, the bridge module could never populate its MDB.
As another example, consider a device acting as router which has received an IP
packet with a TTL of 1. Upon routing the packet the device must send it to the
kernel so that it will route it as well and generate an ICMP Time Exceeded
error datagram. Without letting the kernel route such packets itself, utilities
such as traceroute could never work.
The fundamental ability of sending certain packets to the kernel for processing is called "packet trapping".
The devlink-trap mechanism allows capable device drivers to register their
supported packet traps with devlink and report trapped packets to
devlink for further analysis.
Upon receiving trapped packets, devlink will perform a per-trap packets and
bytes accounting and potentially report the packet to user space via a netlink
event along with all the provided metadata (e.g., trap reason, timestamp, input
port). This is especially useful for drop traps (see :ref:Trap-Types)
as it allows users to obtain further visibility into packet drops that would
otherwise be invisible.
The following diagram provides a general overview of devlink-trap::
Netlink event: Packet w/ metadata
Or a summary of recent drops
^
|
Userspace |
+---------------------------------------------------+
Kernel |
|
+-------+--------+
| |
| drop_monitor |
| |
+-------^--------+
|
| Non-control traps
|
+----+----+
| | Kernel's Rx path
| devlink | (non-drop traps)
| |
+----^----+ ^
| |
+-----------+
|
+-------+-------+
| |
| Device driver |
| |
+-------^-------+
Kernel |
+---------------------------------------------------+
Hardware |
| Trapped packet
|
+--+---+
| |
| ASIC |
| |
+------+
.. _Trap-Types:
The devlink-trap mechanism supports the following packet trap types:
drop: Trapped packets were dropped by the underlying device. Packets
are only processed by devlink and not injected to the kernel's Rx path.
The trap action (see :ref:Trap-Actions) can be changed.exception: Trapped packets were not forwarded as intended by the
underlying device due to an exception (e.g., TTL error, missing neighbour
entry) and trapped to the control plane for resolution. Packets are
processed by devlink and injected to the kernel's Rx path. Changing the
action of such traps is not allowed, as it can easily break the control
plane.control: Trapped packets were trapped by the device because these are
control packets required for the correct functioning of the control plane.
For example, ARP request and IGMP query packets. Packets are injected to
the kernel's Rx path, but not reported to the kernel's drop monitor.
Changing the action of such traps is not allowed, as it can easily break
the control plane... _Trap-Actions:
The devlink-trap mechanism supports the following packet trap actions:
trap: The sole copy of the packet is sent to the CPU.drop: The packet is dropped by the underlying device and a copy is not
sent to the CPU.mirror: The packet is forwarded by the underlying device and a copy is
sent to the CPU.Generic packet traps are used to describe traps that trap well-defined packets or packets that are trapped due to well-defined conditions (e.g., TTL error). Such traps can be shared by multiple device drivers and their description must be added to the following table:
.. list-table:: List of Generic Packet Traps :widths: 5 5 90
source_mac_is_multicastdropvlan_tag_mismatchdropingress_vlan_filterdropingress_spanning_tree_filterdropport_list_is_emptydropport_loopback_filterdropblackhole_routedropttl_value_is_too_smallexceptiontail_dropdropnon_ipdropuc_dip_over_mc_dmacdropdip_is_loopback_addressdropsip_is_mcdropsip_is_loopback_addressdropip_header_corrupteddropipv4_sip_is_limited_bcdropipv6_mc_dip_reserved_scopedropipv6_mc_dip_interface_local_scopedropmtu_value_is_too_smallexceptionunresolved_neighexceptionmc_reverse_path_forwardingexceptionreject_routeexceptionipv4_lpm_missexceptionipv6_lpm_missexceptionnon_routable_packetdropdecap_errorexceptionoverlay_smac_is_mcdropingress_flow_action_dropdropegress_flow_action_dropdropstpcontrollacpcontrollldpcontroligmp_querycontroligmp_v1_reportcontroligmp_v2_reportcontroligmp_v3_reportcontroligmp_v2_leavecontrolmld_querycontrolmld_v1_reportcontrolmld_v2_reportcontrolmld_v1_donecontrolipv4_dhcpcontrolipv6_dhcpcontrolarp_requestcontrolarp_responsecontrolarp_overlaycontrolipv6_neigh_solicitcontrolipv6_neigh_advertcontrolipv4_bfdcontrolipv6_bfdcontrolipv4_ospfcontrolipv6_ospfcontrolipv4_bgpcontrolipv6_bgpcontrolipv4_vrrpcontrolipv6_vrrpcontrolipv4_pimcontrolipv6_pimcontroluc_loopbackcontrollocal_routecontrolexternal_routecontrolipv6_uc_dip_link_local_scopecontrolipv6_dip_all_nodescontrolipv6_dip_all_routerscontrolipv6_router_solicitcontrolipv6_router_advertcontrolipv6_redirectcontrolipv4_router_alertcontrolipv6_router_alertcontrolptp_eventcontrolptp_generalcontrolflow_action_samplecontrolflow_action_trapcontrolearly_dropdropvxlan_parsingdropllc_snap_parsingdropvlan_parsingdroppppoe_ppp_parsingdropmpls_parsingdroparp_parsingdropip_1_parsingdropip_n_parsingdropgre_parsingdropudp_parsingdroptcp_parsingdropipsec_parsingdropsctp_parsingdropdccp_parsingdropgtp_parsingdropesp_parsingdropblackhole_nexthopdropdmac_filterdropeapolcontrollocked_portdropDevice drivers can register driver-specific packet traps, but these must be clearly documented. Such traps can correspond to device-specific exceptions and help debug packet drops caused by these exceptions. The following list includes links to the description of driver-specific traps registered by various device drivers:
.. _Generic-Packet-Trap-Groups:
Generic packet trap groups are used to aggregate logically related packet
traps. These groups allow the user to batch operations such as setting the trap
action of all member traps. In addition, devlink-trap can report aggregated
per-group packets and bytes statistics, in case per-trap statistics are too
narrow. The description of these groups must be added to the following table:
.. list-table:: List of Generic Packet Trap Groups :widths: 10 90
l2_dropsl3_dropsl3_exceptionsbuffer_dropstunnel_dropsacl_dropsstplacplldpmc_snoopingdhcpneigh_discoverybfdospfbgpvrrppimuc_loopbackuc_loopback). This trap is singled-out because in cases such as
one-armed router it will be constantly triggered. To limit the impact on
the CPU usage, a packet trap policer with a low rate can be bound to the
group without affecting other trapslocal_deliveryipv4_bgp)external_deliveryipv6ptp_eventptp_generalacl_sampleacl_trapparser_error_dropseapolAs previously explained, the underlying device can trap certain packets to the CPU for processing. In most cases, the underlying device is capable of handling packet rates that are several orders of magnitude higher compared to those that can be handled by the CPU.
Therefore, in order to prevent the underlying device from overwhelming the CPU, devices usually include packet trap policers that are able to police the trapped packets to rates that can be handled by the CPU.
The devlink-trap mechanism allows capable device drivers to register their
supported packet trap policers with devlink. The device driver can choose
to associate these policers with supported packet trap groups (see
:ref:Generic-Packet-Trap-Groups) during its initialization, thereby exposing
its default control plane policy to user space.
Device drivers should allow user space to change the parameters of the policers (e.g., rate, burst size) as well as the association between the policers and trap groups by implementing the relevant callbacks.
If possible, device drivers should implement a callback that allows user space to retrieve the number of packets that were dropped by the policer because its configured policy was violated.
See tools/testing/selftests/drivers/net/netdevsim/devlink_trap.sh for a
test covering the core infrastructure. Test cases should be added for any new
functionality.
Device drivers should focus their tests on device-specific functionality, such as the triggering of supported packet traps.