libpcap/pcap-sita.html
| A "Distributed Pcap" for
Remote Monitoring LANs & WANs
(Design Notes for the SITA ACN device) | Fulko Hew
SITA INC Canada, Inc.
Revised: October 2, 2007 |
Note: This document is part of the libpcap Git and was derived from 'pcap.3' (circa Aug/07).
The ACN provides a customized/distributed version of this library that allows SMPs to interact with the various IOPs within the site providing a standard mechanism to capture LAN and WAN message traffic.
| SMP | The Supervisory Management Processor where Wireshark (or equivalent) runs in conjunction with a libpcap front-end. | | IOP | I/O Processors where the monitored ports exist in conjunction with a custom device driver/libpcap back-end. |
Each IOP will be capable of supporting multiple connections from an SMP enabling monitoring of more than one interface at a time, each through its own separate connection. The IOP is responsible to ensure and report an error if any attempt is made to monitor the same interface more than once.
There are three applications that will be supported by the ACN version of libpcap. They each use a slightly different mode for looping/capturing and termination as summarized in the following table:
| Application | Capture | Termination |
|---|---|---|
| wireshark | pcap_dispatch(all packets in one buffer of capture only) | pcap_breakloop() |
| tshark | pcap_dispatch(one buffer of capture only) | Since a CTRL-C was used to terminate the application, pcap_breakloop() is never called. |
| tcpdump | pcap_loop(all packets in the next buffer, and loop forever) | pcap_breakloop() |
Note: In all cases, the termination of capturing is always (apparently) followed by pcap_close(). Pcap_breakloop() is only used to stop/suspend looping/processing, and upon close interpretation of the function definitions, it is possible to resume capturing following a pcap_breakloop() without any re-initialization.
The following list of functions is the sub-set of Pcap functions that have been altered/enhanced to support the ACN remote monitoring facility. The remainder of the Pcap functions continue to perform their duties un-altered. Libpcap only supports this mode of operation if it has been configured/compiled for SITA/ACN support.
pcap_findalldevs
pcap_freealldevs
pcap_open_live
pcap_close
pcap_setfilter
pcap_dispatch
pcap_loop
pcap_next
pcap_next_ex
pcap_stats
These subroutines have been modified for the ACN specific distributed and remote monitoring ability perform the following basic functions. More detail is provided in the "SMP/IOP Inter-Process Communication Protocol" section.
| pcap_open_live() | Used to obtain a packet capture descriptor to look at packets on the network. | |
| SMP -> IOP | The SMP will open a connection to the selected IOP on its 'sniffer' port to ensure it is available. It sends a null terminated string identifying the interface to be monitored. | | IOP -> SMP | After any required processing is complete, the IOP will return a null terminated string containing an error message if one occurred. If no error occurred, a empty string is still returned. Errors are:
|
| | pcap_findalldevs() | It constructs a list of network devices that can be opened with pcap_open_live(). | |
| SMP | It obtains a list of IOPs currently available (via /etc/hosts). | | SMP -> IOP | The SMP will sequentially open a connection to each IOP on its 'sniffer' port to ensure the IOP is available. It sends a null terminated empty interface ID followed by the query request command. | | IOP -> SMP | The IOP returns an error response and its list of devices. | | SMP -> IOP | The SMP closes the TCP connection with each IOP. | | SMP | The SMP adds the received information to its internal structure. |
| | pcap_freealldevs() | Used to free a list allocated by pcap_findalldevs(). | |
| SMP | The SMP frees the structure it built as a result of the previous invocation of pcap_findalldevs(). |
| | pcap_dispatch() | Used to collect and process packets. | |
| SMP -> IOP | On the first invocation of pcap_dispatch(), pcap_loop(), or pcap_next(), or pcap_next_ex() following a pcap_open_live(), the SMP will pass down the monitor start command and various parameters the IOP should use. | | IOP -> SMP | The IOP now sends a stream of captured data. | | SMP | The SMP will read the reverse channel of the connection between the SMP and the IOP that provides the captured data (via 'p->read_op' which is 'pcap_read_linux()' until the select() call returns a 'no more data' indication. It will the process (at most) the next 'cnt' packets and invoke the specified callback function for each packet processed. | | IOP | The IOP continues to listen for additional commands as well as capturing and forwarding data to the SMP. |
| | pcap_loop() | Is similar to pcap_dispatch() except it keeps reading packets until the requested number of packets are processed or an error occurs. | |
| SMP -> IOP | On the first invocation of pcap_dispatch(), pcap_loop(), or pcap_next(), or pcap_next_ex() following a pcap_open_live(), the SMP will pass down the monitor start command and various parameters the IOP should use. | | IOP -> SMP | The IOP now sends a stream of captured data. | | SMP | The SMP continuously reads the next packet from the reverse channel of the connection between the SMP and the IOP that provides the captured data (via 'p->read_op' which is 'pcap_read_linux()' until 'cnt' packets have been received. The specified callback function will be invoked for each packet received. | | IOP | The IOP continues to listen for additional commands as well as capturing and forwarding data to the SMP. |
| | pcap_next() | It reads the next packet (by calling pcap_dispatch() with a count of 1) and returns a pointer to the data in that packet. | |
| SMP -> IOP | On the first invocation of pcap_dispatch(), pcap_loop(), or pcap_next(), or pcap_next_ex() following a pcap_open_live(), the SMP will pass down the monitor start command and various parameters the IOP should use. | | IOP -> SMP | The IOP now sends a stream of captured data. | | SMP | The SMP reads only the next packet from the reverse channel of the connection between the SMP and the IOP that provides the captured data (via calling pcap_dispatch() with a count of 1) and returns a pointer to that data by invoking an internal callback. | | IOP | The IOP continues to listen for additional commands as well as capturing and forwarding data to the SMP. |
| | pcap_next_ex() | Reads the next packet and returns a success/failure indication. | |
| SMP -> IOP | On the first invocation of pcap_dispatch(), pcap_loop(), or pcap_next(), or pcap_next_ex() following a pcap_open_live(), the SMP will pass down the monitor start command and various parameters the IOP should use. | | IOP -> SMP | The IOP now sends a stream of captured data. | | SMP | The SMP reads only the next packet from the reverse channel of the connection between the SMP and the IOP that provides the captured data (via calling pcap_dispatch() with a count of 1) and returns separate pointers to both the packet header and packet data by invoking an internal callback. | | IOP | The IOP continues to listen for additional commands as well as capturing and forwarding data to the SMP. |
| | pcap_setfilter() | Used to specify a filter program. | |
| SMP -> IOP | The SMP sends a 'set filter' command followed by the BPF commands. | | IOP -> SMP | The IOP returns a null terminated error string if it failed to accept the filter. If no error occurred, then a NULL terminated empty string is returned instead. Errors are:
|
| | pcap_stats() | Fills in a pcap_stat struct with packet statistics. | |
| SMP -> IOP | The SMP sends a message to the IOP requesting its statistics. | | IOP -> SMP | The IOP returns the statistics. | | SMP | The SMP fills in the structure provided with the information retrieved from the IOP. |
| | pcap_close() | Closes the file and deallocates resources. | |
| SMP -> IOP | The SMP closes the file descriptor, and if the descriptor is that of the communication session with an IOP, it too is terminated. | | IOP | If the IOP detects that its communication session with an SMP has closed, it will terminate any monitoring in progress, release any resources and close its end of the session. It will not maintain persistence of any information or prior mode of operation. |
|
Communications between an SMP and an IOP consists of a TCP session between an ephemeral port on the SMP and the well known port of 49152 (which is the first available port in the 'dynamic and/or private port' range) on an IOP.
Following a TCP open operation the IOP receives a null terminated 'interface ID' string to determine the type of operation that follows:
Every command received by an IOP implies a 'stop trace/stop forwarding' operation must occur before executing the received command.
A session is closed when the SMP closes the TCP session with the IOP. Obviously monitoring and forwarding is also stopped at that time. Note: All multi-octet entities are sent in network neutral order.
Each interface within an IOP will be referred to uniquely. Since an currently contains 8 monitorable WAN ports and a monitorable Ethernet port, the naming convention is:
| Interface # | Type | Name |
|---|---|---|
| 1 | WAN | wan0 |
| 2 | WAN | wan1 |
| 3 | WAN | wan2 |
| 4 | WAN | wan3 |
| 5 | WAN | wan4 |
| 6 | WAN | wan5 |
| 7 | WAN | wan6 |
| 8 | WAN | wan7 |
| 9 | Ethernet | eth0 |
| 10 | Ethernet | eth1 |
The format of the trace data that is sent to the SMP follows a portion of the libpcap file format and is summarized here. This format specifies the generic requirements needed to be able to decode packets, but does not cover ACN specifics such as custom MAC addressing and WAN protocol support.
Although a libpcap file begins with a global header followed by zero or more records for each captured packet, trace data sent to the SMP does NOT begin with a global header. A trace sequence looks like this:
| [Packet Header] | [Packet Data] | [Packet Header] | [Packet Data] | [Packet Header] | [Packet Data] | ... |
Each captured packet starts with a header that contains the following values (in network neutral order):
uint32 tv_sec; /* timestamp seconds */
uint32 tv_usec; /* timestamp microseconds */
uint32 caplen; /* number of octets in the following packet */
uint32 len; /* original length of packet on the wire */
| tv_sec | The date and time when this packet was captured. This value is in seconds since January 1, 1970 00:00:00 GMT; this is also known as a UN*X time_t. You can use the ANSI C time() function from time.h to get this value, but you might use a more optimized way to get this timestamp value. | | tv_usec | The microseconds when this packet was captured, as an offset to ts_sec. Beware: this value must never reach 1 second (1,000,000), in this case ts_sec must be increased instead! | | caplen | The number of bytes actually provided in the capture record. This value should never become larger than len or the snaplen value specified during the capture. | | len | The length of the packet "on the wire" when it was captured. If caplen and len differ, the actually saved packet size was limited by the value of snaplen specified during one of the capture directives such as pcap_dispatch(). |
The actual packet data will immediately follow the packet header as a sequence of caplen octets. Depending on the DLT encoding number assigned to the interface, the packet data will contain an additional custom header used to convey WAN port related information.
PCAP, Wireshark and Tcpdump enhancements have been added to the ACN to support monitoring of its ports, however each of these facilities were focused on capturing and displaying traffic from LAN interfaces. The SITA extensions to these facilities are used to also provide the ability to capture, filter, and display information from an ACN's WAN ports.
Although each packet follows the standard libpcap format, since there are two types of interfaces that can be monitored, the format of the data packet varies slightly.
- For Ethernet (like) devices, the packet format is unchanged from the standard Pcap format.
- For WAN devices, the packet contains a 5 byte header that precedes the actual captured data described by the following table:
| Octet | Name | Mask/Value | Definition |
|---|---|---|---|
| 0 | Control / Status | xxxxxxx0 | Transmitted by capture device |
| xxxxxxx1 | Received by capture device | ||
| 1xxxxxxx | No buffer was available during capture of previous packet. | ||
| 1 | Signals | xxxxxxx1 | DSR asserted |
| xxxxxx1x | DTR asserted | ||
| xxxxx1xx | CTS asserted | ||
| xxxx1xxx | RTS asserted | ||
| xxx1xxxx | DCD asserted | ||
| xx1xxxxx | Undefined | ||
| x1xxxxxx | Undefined | ||
| 1xxxxxxx | Undefined | ||
| 2 | Errors | ||
| (octet 1) | Tx | Rx | |
| --- | --- | --- | --- |
| xxxxxxx1 | Underrun | Framing | |
| xxxxxx1x | CTS Lost | Parity | |
| xxxxx1xx | UART Error | Collision | |
| xxxx1xxx | Re-Tx Limit Reached | Long Frame | |
| xxx1xxxx | Undefined | Short Frame | |
| xx1xxxxx | Undefined | Undefined | |
| x1xxxxxx | Undefined | Undefined | |
| 1xxxxxxx | Undefined | Undefined | |
| 3 | Errors | ||
| (octet 2) | Tx | Rx | |
| --- | --- | --- | --- |
| xxxxxxx1 | Undefined | Non-Octet Aligned | |
| xxxxxx1x | Undefined | Abort Received | |
| xxxxx1xx | Undefined | CD Lost | |
| xxxx1xxx | Undefined | Digital PLL Error | |
| xxx1xxxx | Undefined | Overrun | |
| xx1xxxxx | Undefined | Frame Length Violation | |
| x1xxxxxx | Undefined | CRC Error | |
| 1xxxxxxx | Undefined | Break Received | |
| 4 | Protocol |
| 0x01 | - | LAPB (BOP) | | 0x02 | - | Ethernet 1 | | 0x03 | - | Async (Interrupt IO) | | 0x04 | - | Async (Block IO) | | 0x05 | - | IPARS | | 0x06 | - | UTS | | 0x07 | - | PPP (HDLC) | | 0x08 | - | SDLC | | 0x09 | - | Token Ring 1 | | 0x10 | - | I2C | | 0x11 | - | DPM Link | | 0x12 | - | Frame Relay (BOP) |
Note 1: Ethernet and Token Ring frames will never be sent as DLT_SITA (with the 5 octet header), but will be sent as their corresponding DLT types instead.
|