Network device drivers in Linux

Aapo Kalliola
Aalto University School of Science
Otakaari 1
Espoo, Finland
[email protected]

ABSTRACT that interrupt-driven would be better than polling or vice

In this paper we analyze the interfaces, functionality and versa.
implementation of network device drivers in Linux. We ob-
serve that current real-life network drivers are very heavily In this paper we will examine network device drivers in
optimized compared to a minimum baseline implementation. Linux starting from driver basics and interfaces and finish-
Additionally we analyze some of the optimized functionality ing in a partial analysis of certain functionality in the e1000
in greater detail to gain insight into state-of-the-art network network driver.
device driver implementations.
2. OVERVIEW
Keywords 2.1 Network drivers and interfaces
device driver, network interface, linux A network driver is in principle a simple thing: its purpose is
to provide the kernel with the means to transmit and receive
data packets to and from network using a physical network
1. INTRODUCTION interface. The driver is loaded into the kernel as a network
Devices in Linux are divided into three main categories: module.
block, character and network devices. Block devices pro-
vide aadressable and persistent access to hardware: after Thus the network device driver code interfaces to two direc-
writing to a specific block address subsequent reads from tions: the Linux network stack on one side and device hard-
the same address return the same content that was written. ware on the other. This is illustrated in Figure 1. While the
In contrast, character devices provide unstructured access to network stack interface is hardware-independent the inter-
hardware. Character device may be a separate devices such face to network interface hardware is very much dependent
as a mouse or a modem or part of the internal computer on the hardware implementation. The functionality of the
architecture, for instance a communications bus controller. network interface is accessed through specific memory ad-
dresses, and is thus very much non-generic. For the purposes
Character devices do not typically do buffering while block of this paper we try to remain generic where possible and
devices are accessed through a buffer cache. In some cases go into device-dependent discussion only where absolutely
block devices can also have character device drivers specifi- necessary.
cally for bypassing the cache for raw I/O.
In Linux kernel source tree network driver code for manu-
Network devices in Linux differ significantly from both block facturers is located in linux/drivers/net/ethernet/ [2]. Our
and character devices [1]. Data is typically sent by an up- later analysis of Intel e1000 driver code will be based on the
per program layer by placing it first in a socket buffer and code that is in linux/drivers/net/ethernet/intel/e1000/.
then initiating the packet transmission with a separate func-
tion call. Packet reception may be interrupt-driven, polling-
driven or even alternate between these modes depending on
2.2 Common data structures and functions
The essential data structures and functions for a network
configuration and traffic profile. An optimized driver wants
driver.
to avoid swamping the CPU with interrupts on heavy traf-
fic but the driver also wants to maintain low latency in low
A network interface is described in Linux as a struct net device
traffic scenarios. Therefore it cannot be categorically stated
item. This structure is defined in <linux/netdevice.h>. TODO:
analyze the most interesting (and later occurring) features
of the struct

Packets are handled in transmission as socket buffer struc-

tures (struct sk buff ) defined in <linux/skbuff.h>.

3. DRIVERS IN PRACTISE
In this section we look into network device drivers in the real
world. First we look into the minimal functionality a net-
 Connectivity 3.2 e1000
TODO: Continue to do a more thorough and comprehen-
System call interface
sive overview of the e1000 code and walkthroughs of key
functions.
Kernel
Networking
subsystem
3.2.1 struct e1000_adapter
The structure struct e1000 adapter in e1000.h is the board-
Network specific private data structure. It contains many interesting
subsystem fields for features related to VLANs, link speed and duplex
Software management, packet counters, interrupt throttle rate, TX
support and RX buffers, and also the operating system defined struc-
Interface tures net device and pci dev. An adapter may have multiple
driver
RX and TX queues active at the same time.

Network 3.2.2 e1000_open

Hardware
interface When an e1000 network interface is activated by the system
the function e1000 open is called. This function allocates
resources for transmit and receive operations, registers the
Figure 1: Device driver role in network connectivity interrupt (IRQ) handler with the operating system, starts
a watchdog task and notifies the stack that the interface is
ready. Since e1000 driver uses NAPI it also calls napi enable
before enabling the adapter IRQ.
work driver needs to have to transmit and receive packets.
Then we look at the e1000 driver for Intel network inter-
face cards and summarize its code organization. Finally we 3.2.3 Buffers
look to into the optimizations and other code that the e1000 e1000 has receive (RX) and transmit (TX) buffers imple-
driver has in addition to the bare minimum functionality. mented as ring buffers in e1000.h.

The TX buffer e1000 tx ring has a varying size, while the

3.1 Simple driver e1000 rx ring also contains a cpu value. TODO: purpose?
A minimal network device driver needs to be able to transmit
and receive packets. In order to be able to do this the driver
needs to do a series of tasks: the physical device needs to be
3.3 Simple driver vs. e1000 driver
TODO: Compare e1000 vs simple driver and try to point
detected, enabled and initialized [3]. After initialization the
out the most glaring/interesting differences.
device can be readied for transmitting and receiving packets.
For all this to be possible we need to have an understanding
It is obvious that the e1000 driver is hugely more complex
of the network device, bus-independent device access, PCI
than the simple driver: whereas the simple driver can be
configuration space and the network device chipset’s trans-
implemented in under 1000 lines of code (LOC) the e1000
mission and receive mechanisms.
driver has closer to 20.000 LOC.
For device detection it is possible to use the pci find device e1000 simple name description
function with our device manufacturer’s vendor ID and de- X - net device OS network device struct
vice ID. This gives us a pointer to the device. The device X X pci dev PCI device
can then be enabled with pci enable device using the pointer - X base I/O address.
as the only argument. X X TX and RX buffers
X - Device statistics
TODO: some more content about bus-independent memory
X - active vlans Virtual LAN
access and PCI
X - wol Wake-on-LAN
As previously discussed, network interfaces are seen by the X - tx itr, . . . Interrupt throttling
Linux networking stack as devices in net device. This struc- X - Link speed control
ture is then updated with actual device values during device X - HW checksumming
initialization. In initialization the driver retrieves the base X - NAPI
memory address for the device registers, which is needed in X - Thread safety
all subsequent addressing of device registers. X - Watchdog functions

The transmission and receiving mechanisms on hardware Table 1: Adapter private structure comparison
are device-dependent and can be discovered either through
manufacturer documentation such as [4] or by reverse engi- One comparison that can be done to get some insight about
neering. Usual areas of interest are the transmit and receive the feature differences between the simple driver and the
memory buffers. These buffers can be implemented on hard- e1000 driver is to look into their private data structure and
ware as, for instance, ring buffers. see what fields they contain. These names are shown, if
applicable, and grouped into categories with descriptions in
TODO: more details about the receive and transmit control Table 1.
4. FEATURES and polling vs. interrupt driven mode selection depending
In this section we look into specific feature implementations on traffic profile. This effectively places a significant amount
in the e1000 driver. of burden on the network interface hardware and driver as
weak implementation in them might well hamper the per-
4.1 Interrupt throttling formance of the whole system.
When the network interface receives a series of packet it
can send an interrupt, buffer the packet and wait for a poll Overall our still incomplete analysis of a minimal network
later on, or buffer the packet and send an interrupt when device driver and the state-of-the-art e1000 driver our pa-
the allocated buffer size hits a given threshold. This set of per shows that the current real-life drivers are thoroughly
different behaviours is an area that has plenty of opportuni- optimized and feature-rich.
ties for optimization for different traffic profiles. Polling is
typically less resource-consuming with heavy traffic, whereas 6. REMAINING WORK...
interrupt-driven operation is better for low latencies when This paper is still very much a work in progress, apologies to
the traffic is not very heavy. the reviewers for this. I’ll expand on at least simple driver
vs e1000 and features analysis, but in general more details
The e1000 driver has an internal interrupt throtte rate (ITR) and verification of content currentness (most sources are >5
control mechanism that seeks to dynamically adjust the in- years old) will be forthcoming.
terrupt frequency depending on the current traffic pattern.
This functionality is controlled using the InterruptThrot- 7. REFERENCES
tleRate parameter in e1000 param.c. Functions relevant to [1] J. Corbet, A. Rubini, and G. Kroah-Hartman. Linux
ITR are: Device Drivers, Third Edition. 2005.
• e1000 update itr [2]
• e1000 set itr http://lxr.linux.no/#linux+v3.6.7/drivers/net/ethernet,
November 2012.
• e1000 clean (NAPI related)
[3] M. L. Jangir. Writing network device drivers for linux.
It seems that the driver is capable of adjusting not only Linux Gazette, (156), November 2008.
the interrupt frequency but also whether it works in polling [4] Realtek. RTL8139(A/B) Programming guide: (V0.1).
or interrupt-driven mode (e1000 clean). Investigate fur- 1999.
ther.

TODO: further analyze the state machine and the input

parameters

4.2 Protocol offloading

TODO: Checksumming discussion.

TCP Segmentation offloading (TSO) is the functionality that

does the the cutting of packets to a maximum of 1500 byte
chunks on transmission. The purpose of doing this in the
network interface hardware is to lessen the load of the system
CPU while maintaining high throughput. Since the operat-
ing system must know if the network interface supports TSO
the support is defined by the e1000 driver to the hardware
feature of the net device structure. TODO: continue this..

TCP and UDP checksum offloading is enabled for packet

reception in e1000 configure rx. Actual RX checksum cal-
culation is done in e1000 rx checksum. TX checksums are
unsurprisingly done in e1000 tx csum. TODO: does it gen-
erate a tx checksum for UDP??

5. CONCLUSION
Modern network device driver and hardware handles func-
tionality that has originally been the responsibility of the
operating system and the CPU, such as packet fragmenta-
tion and checksum computation. This leads to significant
complexity in the driver compared to a more feature-limited
approach.

One of the interesting findings in this paper was that the

e1000 driver has significant control over the packet process-
ing mode: it is capable of adjusting its interrupt generation