On a NIC's operating system, schedulers and high-performance networking applications

Two critical issues impact the overall performance of Linux clusters based on Intel servers: inter-process communication latency and data throughput. Underlying both of these performance challenges is the inefficient use of computational power and server CPU cycles to process the network protocols. Today's modern high-end Network Interface Cards (NICs) are equipped with an onboard CPU. In most cases, these CPU's are only used by the vendor and are operated by a proprietary OS, which makes them inaccessible to the HPC application developer. In this paper we present a design and implementation of a framework for building high-performance networking applications. The framework consists of an embedded NIC Operating System with a specialized scheduler. The main challenge in developing such a scheduler is the lack of a preemption mechanism in most high-end NICs. Our scheduler provides finer-grained schedules than the alternatives. We have implemented several network applications, and were able to increase their throughput while decreasing the host's CPU utilization.