Myricom logotype

SNF API Reference

API Reference for SNF and usage model. More...

API Reference

 Receive-Side Scaling (RSS)
 Device flags for process-sharing, packet duplication

Data Structures

struct  snf_ifaddrs
struct  snf_recv_req
struct  snf_ring_stats

Defines

#define SNF_VERSION_API   0x0002

Typedefs

typedef snf_handle * snf_handle_t
 opaque snf device handle
typedef snf_ring * snf_ring_t
 opaque snf ring handle

Functions

int snf_init (uint16_t api_version)
 Initialize Sniffer Library with api_version == SNF_VERSION_API.
int snf_getifaddrs (struct snf_ifaddrs **ifaddrs_o)
void snf_freeifaddrs (struct snf_ifaddrs *ifaddrs)
int snf_open (uint32_t boardnum, int num_rings, const struct snf_rss_params *rss_parms, int64_t dataring_sz, int flags, snf_handle_t *devhandle)
 Open a device for single or multi-ring operation.
int snf_start (snf_handle_t devhandle)
int snf_stop (snf_handle_t devhandle)
int snf_close (snf_handle_t devhandle)
 Close board.
int snf_ring_open (snf_handle_t devhandle, snf_ring_t *ringh)
int snf_ring_close (snf_ring_t ringh)
int snf_ring_recv (snf_ring_t ringh, int timeout_ms, struct snf_recv_req *recv_req)
 Receive next packet from a receive ring.
int snf_ring_getstats (snf_ring_t ringh, struct snf_ring_stats *stats)
 Get statistics from a receive ring.

Detailed Description

API Reference for SNF and usage model.

API Reference

The Sniffer API model is best summarized in the following steps:

Processing Model

The API promotes a processing model mindful of the multiple cores available on today's systems by enabling packets to be received through multiple receive rings. This feature appears in many 10GBit Ethernet drivers as a Receive-Side Scaling (RSS) but instead of being an feature internal to the driver, the Sniffer API presents the rings (or queues) as a first-order interface to the user. Instead of internally maintaining multiple rings as part of the driver, the Sniffer API allows users to directly create rings and retain a better control over how rings are allocated on a typical multicore system. The flexibility over ring allocation is balanced with a receive function with much stricter semantics, but only to pursue these goals:

Multiple Receive Rings

The number of requested receive rings is part of the function call to open a device for packet capture. While users retain control over how many rings are to be allocated by the API, the API best matches processing models that allocate one ring per core, where each core will both capture packets from Sniffer and process/analyze them in place. In-place processing refers is an effort to contain the packet capture and analysis within a CPU core's memory domain, which includes all levels of caches and RAM closest to the core. Since today's multicore platforms are increasingly complex in the non-uniformity of memory access (NUMA), it is best to minimize the amount of memory accesses across memory domains and to maximize the amount of work that can be done by one core within its own memory domain. As such, Sniffer assumes that for best performance, users recognize the importance of opening rings and maintain strong ties between each ring's single consuming thread and the core where it is scheduled.

Receive Side Scaling Modes

When multiple rings are used, users can specify how packets should be partitioned based on a set of RSS flags. These flags instruct how Sniffer should hash incoming packets such that per-flow affinity is maintained in the same ring if so desired. This effectively allows each ring to virtualize the underlying network interface by ensuring that packets within the same flow are deterministically delivered to the same receive ring. Users should not need to reconstruct the ordering of flows even though incoming data is being split across many receive rings.

Zero-Copy receives

When opening a device with one or multiple rings, users can also specify combined amount of memory that can be consumed by all rings to store packet data. If left unspecified, the implementaiton will default to a chosing a relatively conservative amount of memory, assuming that consumers can process incoming packets at line rate. Resizing the ring should be considered only to address specific buffering concerns and when multiple rings are not possible. It is possible that larger rings are necessary to mitigate the non-real-time behavior of some of the supported operating systems. Larger data rings will only serve as a temporary relief for users that cannot consume incoming data at line rates. If the application does not return packets to the data ring as fast as the packets are coming in, the receive ring will eventually overflow and the packets will be dropped.

API Software Overhead

The API tries to minimize software overhead in as many areas that can be addressed directly by the library. This includes duplicating some internal data structures to prevent false sharing between multiple ring consumers. More importantly, however, is the explicit decision to not provide any locking for the main receive function (snf_ring_recv). Even light forms of locking can impact processing rates when incoming data rates are roughly 15 Mpps (or every 70 nanoseconds). Instead, the API promotes the use of multiple rings to improve concurrency in packet processing and leave more breathing room for packet-per-packet analysis on each consuming core.

Thread-Safety

Users should consult the API reference for information on thread safety as each function extensively documents how in can be used in threaded environements. While most API functions provide strong thread safety guarantees, the main receive function (snf_ring_recv) is specifically not thread-safe for software overhead reasons explained in API Software Overhead. The expectation is that multiple threads would each open their own receive rings and independently consume packets on their own rings. Specifically, the Sniffer implementation assumes that with each successive call to snf_ring_recv within the context of a ring always means that the previous packet was consumed by the previous receive on that ring. This should not come as a surprise for users that have been exposed to the serialization present in libpcap-type interfaces. However, this may not necessarily match up well with users that expected to separate their computing resources (i.e. cores) between capture and analysis. These users can always resort to using a single receive ring.

Implementation details

Timestamps

Packet timestamps are available from each packet for packet arrival as seen from the NIC. The frequency of the NIC's clock is 2MHz plus/minus 100ppm. The 64-bit timestamp returned to the user in the receive call is normalized in host nanoseconds ctime and Sniffer internally ensures that both clocks remain synchronized at a regular interval. It should be straightforward for the user to convert nanoseconds to a struct timeval if so be needed.

Library Memory Consumption

In order to efficiently use host memory, data rings are allocated when the device is opened as a pool of 4KB pages. At open time, users can specify the number of data rings that are allocated as well as the amount of data_ring_size that can be allocated for all rings. Internally, the Sniffer library requires an additional 1/32nd of the data_ring_size for each ring and some additional inter-ring synchronization memory. This can be summarized as at most 10% of the data_ring_size is allocated internally by Sniffer.

Ethernet and Sniffer Driver Modes

At any time, irrespective of the underlying device being enabled for Sniffer or not, users can configure a Sniffer-capable device as a regular ethernet device (i.e. typically via ifconfig). Unless the device is opened for capture, both send and receive functionality works as expected from any Ethernet driver. When the device is opened for capture, packets usually destined to the Ethernet driver are delivered to a Sniffer receive ring instead. However, users can still rely on the Ethernet's RAW sockets interface to send packets (a sample raw socket send enabled with Sniffer receive calls is provided as a test).

Define Documentation

#define SNF_VERSION_API   0x0002

SNF API version number (16 bits) Least significant byte increases for minor backwards compatible changes in the API. Most significant byte increases for incompatible changes in the API

0x0002: Add nic_bytes_recv counter to stats to help users calculate the amount of bandwidth that is actually going through the NIC port.

Typedef Documentation

typedef struct snf_handle* snf_handle_t

opaque snf device handle

Opaque snf handle, allocated at snf_open time when a device can be succesfully opened

typedef struct snf_ring* snf_ring_t

opaque snf ring handle

Opaque snf ring handle, allocated at snf_ring_open time when a ring can be succesfully opened

Function Documentation

int snf_close ( snf_handle_t  devhandle  ) 

Close board.

This function can be closed once all opened rings (if any) are closed through snf_ring_close. Once a board is determined to be closable, it is implicitly called as if a call had been previously made to snf_stop.

Return values:
0 Successful.
EBUSY Some rings are still opened and the board cannot be closed (yet).
Postcondition:
If successful, all resources allocated at open time are unallocated and the device switches from Sniffer mode to Ethernet mode such that the Ethernet driver resumes receiving packets.

void snf_freeifaddrs ( struct snf_ifaddrs *  ifaddrs  ) 

Free the list of library allocated memory for snf_getifaddrs

Parameters:
ifaddrs Pointer to ifaddrs allocated via snf_getifaddrs

int snf_getifaddrs ( struct snf_ifaddrs **  ifaddrs_o  ) 

Get a list of Sniffer-capable ethernet devices.

Parameters:
ifaddrs_o Library-allocated list of Sniffer-capable devices
Remarks:
Much like getifaddrs, the user can traverse the list until snf_ifa_next is NULL. The interface will show up if the ethernet driver sees the device but the interface does not have to be brought up with an IP address (i.e. no need to 'ifconfig up').
Postcondition:
User should call snf_freeifaddrs to free the memory that was allocated by the library.

int snf_init ( uint16_t  api_version  ) 

Initialize Sniffer Library with api_version == SNF_VERSION_API.

Initializes the sniffer library.

Parameters:
api_version Must always be SNF_VERSION_API
Remarks:
This must be called before any other call to the sniffer library.

The library may be safely initialized multiple times, although the api_version shoudl be the same SNF_VERSION_API each time.

int snf_open ( uint32_t  boardnum,
int  num_rings,
const struct snf_rss_params *  rss_parms,
int64_t  dataring_sz,
int  flags,
snf_handle_t *  devhandle 
)

Open a device for single or multi-ring operation.

Opens a board for sniffing and allocates a device handle.

Parameters:
boardnum Boards are numbered from 0 to N-1 where 'N' is the number of Myricom ports available on the system. snf_getifaddrs() may be a useful utility to retrieve the board number by interface name or mac address if there are multiple
num_rings Number of rings to allocate for receive-side scaling feature, which determines how many different threads can open their own ring via snf_ring_open(). If set to 0 or less than zero, default value is used unless SNF_NUM_RINGS is set in the environment.
rss_parms Points to a user-initialized structure that selects the RSS mechanism to apply to each incoming packet. This parameter is only meaningful if there are more than 1 rings to be opened. By default, if users pass a NULL value, the implementation will select its own mechanism to divide incoming packets across rings. RSS parameters are documented in Receive-Side Scaling (RSS).
dataring_sz Represents the total amount of memory to be used to store incoming packet data for *all* rings to be opened. If the value is set to 0 or less than 0, the library tries to choose a sensible default unless SNF_DATARING_SIZE is set in the environment. The value can be specified in megabytes (if it is less than 1048576) or is otherwise considered to be in bytes. In either case, the library may slightly adjust the user's request to satisfy alignment requirements (typically 2MB boundaries).
flags A mask of flags documented in Device flags for process-sharing, packet duplication.
devhandle Device handle allocated if the call is successful
Return values:
0 Successful. the board is opened and a value devhandle is allocated (see remarks)
EBUSY Device is already opened
EINVAL Invalid argument passed, most probably num_rings (if not, check syslog)
E2BIG Driver could not allocate requested dataring_sz (check syslog)
ENOMEM Either library or driver did not have enough memory to allocate handle descriptors (but not data ring).
Postcondition:
If succesful, the NIC switches from Ethernet mode to Sniffer mode and the Ethernet driver stops receiving packets.

If successful, a call to snf_start is required to the Sniffer-mode NIC to deliver packets to the host, and this call must occur after at least one ring is opened (snf_ring_open).

int snf_ring_close ( snf_ring_t  ringh  ) 

Close a ring

This function is used to inform the underlying device that no further calls to snf_ring_recv will be made. If the device is not subsequently closed (snf_close), all packets that would have been delivered to this ring are dropped. Also, by calling this function, users confirm that all packet processing for packets obtained on this ring via snf_ring_recv is complete.

Parameters:
ringh Ring handle
Return values:
0 Successful.
Postcondition:
The user has processed the last packet obtained with snf_ring_recv and the device can safely be closed via snf_close if all other rings are also closed.

int snf_ring_getstats ( snf_ring_t  ringh,
struct snf_ring_stats *  stats 
)

Get statistics from a receive ring.

Parameters:
ringh Ring handle
stats User-provided pointer to a statistics structure snf_ring_stats, filled in by the library.
Remarks:
This call is provided as a convenience and should not be relied on for time-critical applications or for high levels of accuracy. Statistics are only updated by the NIC periodically.
Warning:
Administrative clearing of NIC counters while a Sniffer-based application is running will cause some of the ring counters to be incorrect.

int snf_ring_open ( snf_handle_t  devhandle,
snf_ring_t *  ringh 
)

Opens a ring from an opened board.

Parameters:
devhandle Device handle, obtained from a successful call to snf_ring_open
ringh Ring handle allocated if the call is successful.
Return values:
0 Successful. The ring is opened and ringh contains the ring handle.
EBUSY Too many rings already opened
Postcondition:
If successful, a call to snf_start is required to the Sniffer-mode NIC to deliver packets to the host.

int snf_ring_recv ( snf_ring_t  ringh,
int  timeout_ms,
struct snf_recv_req *  recv_req 
)

Receive next packet from a receive ring.

This function is used to return the next available packet in a receive ring. The function can block indefinitely, for a specific timeout or be used as a non-blocking call with a timeout of 0.

Parameters:
ringh Ring handle (from snf_ring_open)
timeout_ms Receive timeout to control how the function blocks for the next packet. If the value is less than 0, the function can block indefinitely. If the value is 0, the function is guaranteed to never enter a blocking state and returns EAGAIN unless there is a packet waiting. If the value is greater than 0, the caller inidicates a desired wait time in milliseconds. With a non-zero wait time, the function only blocks if there are no outstanding packets. If the timeout expires before a packet can be received, the function returns EAGAIN (and not ETIMEDOUT). In all cases, users should expect that the function may return EINTR as the result of an interrupt.
recv_req Receive Packet structure, only updated when a the function returns 0 for a successful packet receive (snf_recv_req)
Return values:
0 Successful packet delivery, recv_req is updated with packet information.
EINTR The call was interrupted by a signal handler
EAGAIN No packets available (only when timeout is >= 0).
Remarks:
The packet returned always points directly into the receive ring where the NIC has DMAed the packet (there are no copies). As such, the user obtains a pointer to library/driver allocated memory. Users can modify the contents of the packets but should remain within the boundaries of pkt_addr and length.

Upon calling the function, the library assumes that the user is done processing the previous packet. The same assumption is made when the ring is closed (snf_ring_close).

int snf_start ( snf_handle_t  devhandle  ) 

Start packet capture on a board. Packet capture is only started if it is currently stopped or not yet started for the first time.

Parameters:
devhandle Device handle
Remarks:
It is safe to restart packet capture via snf_start and snf_stop.

This call must be called before any packet can be received.

int snf_stop ( snf_handle_t  devhandle  ) 

Stop packet capture on a board. This function should be used carefully in multi-process mode as a single stop command stops packet capture on all rings. It is usually best to simply snf_ring_close a ring to stop capture on a ring.

Parameters:
devhandle Device handle
Remarks:
Stop instructs the NIC to drop all packets until the next snf_start() or until the board is closed. The NIC only resumes delivering packets when the board is closed, not when traffic is stopped.

Myricom banner
26 October 2010 Sniffer10G 1.0.2_pcap1.1.1