I'm stuck on problem where my machine started to drops packets with no sign of ANY system load or high interrupt usage after an upgrade to Ubuntu 12.04. My server is a network monitoring sensor, running Ubuntu LTS 12.04, it passively collects packets from 5 interfaces doing network intrusion type stuff. Before the upgrade I managed to collect 200+GB of packets a day while writing them to disk with around 0% packet loss depending on the day with the help of CPU affinity and NIC IRQ to CPU bindings. Now I lose a great deal of packets with none of my applications running and at very low PPS rate which a modern workstation NIC would have no trouble with.

Specs: x64 Xeon 4 cores 3.2 Ghz 16 GB RAM NICs: 5 Intel Pro NICs using the e1000 driver (NAPI). [1] eth0 and eth1 are integrated NICs (in the motherboard) There are 2 other PCI-X network cards, each with 2 Ethernet ports.

3 of the interfaces are running at Gigabit Ethernet, the others are not because they're attached to hubs.

Specs: [2] http://support.dell.com/support/edocs/systems/pe2850/en/ug/t1390aa.htm

uptime
 17:36:00 up  1:43,  2 users,  load average: 0.00, 0.01, 0.05

# uname -a
Linux nms 3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64 x86_64     x86_64 GNU/Linux

I also have the CPU governor set to performance mode and irqbalance off. The problem still occurs with them on.

# lspci -t -vv
-[0000:00]-+-00.0  Intel Corporation E7520 Memory Controller Hub
           +-02.0-[01-03]--+-00.0-[02]----0e.0  Dell PowerEdge Expandable RAID controller 4
           |               \-00.2-[03]--
           +-04.0-[04]--
           +-05.0-[05-07]--+-00.0-[06]----07.0  Intel Corporation 82541GI Gigabit Ethernet Controller
           |               \-00.2-[07]----08.0  Intel Corporation 82541GI Gigabit Ethernet Controller
           +-06.0-[08-0a]--+-00.0-[09]--+-04.0  Intel Corporation 82546EB Gigabit Ethernet Controller (Copper)
           |               |            \-04.1  Intel Corporation 82546EB Gigabit Ethernet Controller (Copper)
           |               \-00.2-[0a]--+-02.0  Digium, Inc. Wildcard TE210P/TE212P dual-span T1/E1/J1 card 3.3V
           |                            +-03.0  Intel Corporation 82546EB Gigabit Ethernet Controller (Copper)
           |                            \-03.1  Intel Corporation 82546EB Gigabit Ethernet Controller (Copper)
           +-1d.0  Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #1
           +-1d.1  Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #2
           +-1d.2  Intel Corporation 82801EB/ER (ICH5/ICH5R) USB UHCI Controller #3
           +-1d.7  Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller
           +-1e.0-[0b]----0d.0  Advanced Micro Devices [AMD] nee ATI RV100 QY [Radeon 7000/VE]
           +-1f.0  Intel Corporation 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge
           \-1f.1  Intel Corporation 82801EB/ER (ICH5/ICH5R) IDE Controller

I believe the NIC nor the NIC drivers are dropping the packets because ethtool reports 0 under rx_missed_errors and rx_no_buffer_count for each interface. On the old system, if it couldn't keep up this is where the drops would be. I drop packets on multiple interfaces just about every second, usually in small increments of 2-4.

I tried all these sysctl values, I'm currently using the uncommented ones.

# cat /etc/sysctl.conf
# high
net.core.netdev_max_backlog = 3000000
net.core.rmem_max = 16000000
net.core.rmem_default = 8000000
# defaults
#net.core.netdev_max_backlog = 1000
#net.core.rmem_max = 131071
#net.core.rmem_default = 163480
# moderate
#net.core.netdev_max_backlog = 10000
#net.core.rmem_max = 33554432
#net.core.rmem_default = 33554432

Here's an example of an interface stats report with ethtool. They are all the same, nothing is out of the ordinary ( I think ), so I'm only going to show one:

ethtool -S eth2
NIC statistics:
     rx_packets: 7498
     tx_packets: 0
     rx_bytes: 2722585
     tx_bytes: 0
     rx_broadcast: 327
     tx_broadcast: 0
     rx_multicast: 1504
     tx_multicast: 0
     rx_errors: 0
     tx_errors: 0
     tx_dropped: 0
     multicast: 1504
     collisions: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_errors: 0
     rx_no_buffer_count: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_window_errors: 0
     tx_abort_late_coll: 0
     tx_deferred_ok: 0
     tx_single_coll_ok: 0
     tx_multi_coll_ok: 0
     tx_timeout_count: 0
     tx_restart_queue: 0
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     rx_align_errors: 0
     tx_tcp_seg_good: 0
     tx_tcp_seg_failed: 0
     rx_flow_control_xon: 0
     rx_flow_control_xoff: 0
     tx_flow_control_xon: 0
     tx_flow_control_xoff: 0
     rx_long_byte_count: 2722585
     rx_csum_offload_good: 0
     rx_csum_offload_errors: 0
     alloc_rx_buff_failed: 0
     tx_smbus: 0
     rx_smbus: 0
     dropped_smbus: 01

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:11:43:e0:e2:8c  
          UP BROADCAST RUNNING NOARP PROMISC ALLMULTI MULTICAST  MTU:1500  Metric:1
          RX packets:373348 errors:16 dropped:95 overruns:0 frame:16
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:356830572 (356.8 MB)  TX bytes:0 (0.0 B)

eth1      Link encap:Ethernet  HWaddr 00:11:43:e0:e2:8d  
          UP BROADCAST RUNNING NOARP PROMISC ALLMULTI MULTICAST  MTU:1500  Metric:1
          RX packets:13616 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:8690528 (8.6 MB)  TX bytes:0 (0.0 B)

eth2      Link encap:Ethernet  HWaddr 00:04:23:e1:77:6a  
          UP BROADCAST RUNNING NOARP PROMISC ALLMULTI MULTICAST  MTU:1500  Metric:1
          RX packets:7750 errors:0 dropped:471 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2780935 (2.7 MB)  TX bytes:0 (0.0 B)

eth3      Link encap:Ethernet  HWaddr 00:04:23:e1:77:6b  
          UP BROADCAST RUNNING NOARP PROMISC ALLMULTI MULTICAST  MTU:1500  Metric:1
          RX packets:5112 errors:0 dropped:206 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:639472 (639.4 KB)  TX bytes:0 (0.0 B)

eth4      Link encap:Ethernet  HWaddr 00:04:23:b6:35:6c  
          UP BROADCAST RUNNING NOARP PROMISC ALLMULTI MULTICAST  MTU:1500  Metric:1
          RX packets:961467 errors:0 dropped:935 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:958561305 (958.5 MB)  TX bytes:0 (0.0 B)

eth5      Link encap:Ethernet  HWaddr 00:04:23:b6:35:6d  
          inet addr:192.168.1.6  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4264 errors:0 dropped:16 overruns:0 frame:0
          TX packets:699 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:572228 (572.2 KB)  TX bytes:124456 (124.4 KB)

I tried the defaults, then started to play around with settings. I wasn't using any flow control and I increased the RxDescriptor count to 4096 before the upgrade as well without any problems.

# cat /etc/modprobe.d/e1000.conf
options e1000 XsumRX=0,0,0,0,0 RxDescriptors=4096,4096,4096,4096,4096 FlowControl=0,0,0,0,0 debug=16

Here's my network configuration file, I turned off checksumming and various offloading mechanisms along with setting CPU affinity with heavy use interfaces getting an entire CPU and light use interfaces sharing a CPU. I used these settings prior to the upgrade without problems.

# cat /etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface

auto eth0 
iface eth0 inet manual
    pre-up /sbin/ethtool -G eth0 rx 4096 tx 0
    pre-up /sbin/ethtool -K eth0 gro off gso off rx off
    pre-up /sbin/ethtool -A eth0 rx off autoneg off
    up ifconfig eth0 0.0.0.0 -arp promisc mtu 1500 allmulti txqueuelen 0 up
    post-up echo "4" > /proc/irq/48/smp_affinity
        down ifconfig eth0 down
        post-down /sbin/ethtool -G eth0 rx 256 tx 256
        post-down /sbin/ethtool -K eth0 gro on gso on rx on
        post-down /sbin/ethtool -A eth0 rx on autoneg on    

auto eth1
iface eth1 inet manual
    pre-up /sbin/ethtool -G eth1 rx 4096 tx 0
    pre-up /sbin/ethtool -K eth1 gro off gso off rx off
    pre-up /sbin/ethtool -A eth1 rx off autoneg off
    up ifconfig eth1 0.0.0.0 -arp promisc mtu 1500 allmulti txqueuelen 0 up
    post-up echo "4" > /proc/irq/49/smp_affinity
        down ifconfig eth1 down
        post-down /sbin/ethtool -G eth1 rx 256 tx 256
        post-down /sbin/ethtool -K eth1 gro on gso on rx on
        post-down /sbin/ethtool -A eth1 rx on autoneg on

auto eth2
iface eth2 inet manual
    pre-up /sbin/ethtool -G eth2 rx 4096 tx 0
    pre-up /sbin/ethtool -K eth2 gro off gso off rx off
    pre-up /sbin/ethtool -A eth2 rx off autoneg off
    up ifconfig eth2 0.0.0.0 -arp promisc mtu 1500 allmulti txqueuelen 0 up
    post-up echo "1" > /proc/irq/82/smp_affinity
    down ifconfig eth2 down
    post-down /sbin/ethtool -G eth2 rx 256 tx 256
    post-down /sbin/ethtool -K eth2 gro on gso on rx on 
    post-down /sbin/ethtool -A eth2 rx on autoneg on

auto eth3
iface eth3 inet manual
    pre-up /sbin/ethtool -G eth3 rx 4096 tx 0
    pre-up /sbin/ethtool -K eth3 gro off gso off rx off
    pre-up /sbin/ethtool -A eth3 rx off autoneg off
    up ifconfig eth3 0.0.0.0 -arp promisc mtu 1500 allmulti txqueuelen 0 up
    post-up echo "2" > /proc/irq/83/smp_affinity
        down ifconfig eth3 down
        post-down /sbin/ethtool -G eth3 rx 256 tx 256
        post-down /sbin/ethtool -K eth3 gro on gso on rx on
        post-down /sbin/ethtool -A eth3 rx on autoneg on

auto eth4
iface eth4 inet manual
    pre-up /sbin/ethtool -G eth4 rx 4096 tx 0
    pre-up /sbin/ethtool -K eth4 gro off gso off rx off
    pre-up /sbin/ethtool -A eth4 rx off autoneg off
    up ifconfig eth4 0.0.0.0 -arp promisc mtu 1500 allmulti txqueuelen 0 up
    post-up echo "4" > /proc/irq/77/smp_affinity
        down ifconfig eth4 down
        post-down /sbin/ethtool -G eth4 rx 256 tx 256
        post-down /sbin/ethtool -K eth4 gro on gso on rx on
        post-down /sbin/ethtool -A eth4 rx on autoneg on

auto eth5
iface eth5 inet static
    pre-up /etc/fw.conf
    address 192.168.1.1
    netmask 255.255.255.0
    broadcast 192.168.1.255
    gateway 192.168.1.1
    dns-nameservers 192.168.1.2 192.168.1.3
    up ifconfig eth5 up
    post-up echo "8" > /proc/irq/77/smp_affinity
    down ifconfig eth5 down

Here's a few examples of packet drops, i ran one after another, probabling totaling 3 or 4 seconds. You can see increases in the drops from the 1st and 3rd. This was a non-busy time, very little traffic.

# awk '{ print $1,$5 }' /proc/net/dev 
Inter-| 
face drop
eth3: 225
lo: 0
eth2: 505
eth1: 0
eth5: 17
eth0: 105
eth4: 1034

# awk '{ print $1,$5 }' /proc/net/dev 
Inter-| 
face drop
eth3: 225
lo: 0
eth2: 507
eth1: 0
eth5: 17
eth0: 105
eth4: 1034

# awk '{ print $1,$5 }' /proc/net/dev 
Inter-| 
face drop
eth3: 227
lo: 0
eth2: 512
eth1: 0
eth5: 17
eth0: 105
eth4: 1039

I tried the pci=noacpi options. With and without, it's the same. This is what my interrupt stats looked like before the upgrade, after, with ACPI on PCI it showed multiple NICs bound to an interrupt and shared with other devices such as USB drives which I didn't like so I think i'm going to keep it with ACPI off as it's easier to designate sole purpose interrupts. Is there any advantage I would have using the default i.e. ACPI w/ PCI. ?

# cat /etc/default/grub | grep CMD_LINE

GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1 noacpi pci=noacpi"
GRUB_CMDLINE_LINUX=""

# cat /proc/interrupts 
            CPU0       CPU1       CPU2       CPU3       
   0:         45          0          0         16   IO-APIC-edge      timer
   1:          1          0          0       7936   IO-APIC-edge      i8042
   2:          0          0          0          0    XT-PIC-XT-PIC    cascade
   6:          0          0          0          3   IO-APIC-edge      floppy
   8:          0          0          0          1   IO-APIC-edge      rtc0
   9:          0          0          0          0   IO-APIC-edge      acpi
  12:          0          0          0       1809   IO-APIC-edge      i8042
  14:          1          0          0       4498   IO-APIC-edge      ata_piix
  15:          0          0          0          0   IO-APIC-edge      ata_piix
  16:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb2
  18:          0          0          0       1350   IO-APIC-fasteoi   uhci_hcd:usb4, radeon
  19:          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
  23:          0          0          0       4099   IO-APIC-fasteoi   ehci_hcd:usb1
  38:          0          0          0      61963   IO-APIC-fasteoi   megaraid
  48:          0          0    1002319          4   IO-APIC-fasteoi   eth0
  49:          0          0      38772          3   IO-APIC-fasteoi   eth1
  77:          0          0     130076     432159   IO-APIC-fasteoi   eth4
  78:          0          0          0      23917   IO-APIC-fasteoi   eth5
  82:    1329033          0          0          4   IO-APIC-fasteoi   eth2
  83:          0    4886525          0          6   IO-APIC-fasteoi   eth3
 NMI:          5          6          4          5   Non-maskable interrupts
 LOC:      61409      57076      64257     114764   Local timer interrupts
 SPU:          0          0          0          0   Spurious interrupts
 IWI:          0          0          0          0   IRQ work interrupts
 RES:      17956      25333      13436      14789   Rescheduling interrupts
 CAL:      22436        607        539        478   Function call interrupts
 TLB:       1525       1458       4600       4151   TLB shootdowns
 TRM:          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0   Machine check exceptions
 MCP:         16         16         16         16   Machine check polls
 ERR:          0
 MIS:          0

Here's sample output of vmstat, showing the system. Barebones system right now.

 root@nms:~# vmstat -S m 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0  14992    192   1029    0    0    56     2  419   29  1  0 99  0
 0  0      0  14992    192   1029    0    0     0     0  922   27  0  0 100  0
 0  0      0  14991    192   1029    0    0     0    36  763   50  0  0 100  0
 0  0      0  14991    192   1029    0    0     0     0  646   35  0  0 100  0
 0  0      0  14991    192   1029    0    0     0     0  722   54  0  0 100  0
 0  0      0  14991    192   1029    0    0     0     0  793   27  0  0 100  0
 ^C

Here's dmesg output. I can't figure out why my PCI-X slots are negotiated as PCI. The network cards are all PCI-X with the exception of the integrated NICs that came with the server. In the output below it looks as if eth3 and eth2 negotiated at PCI-X speeds rather than PCI:66Mhz. Wouldn't they all drop to PCI:66Mhz? If your integrated NICs are PCI, as labeled below (eth0,eth1), then wouldn't all devices on your bus speed drop down to that slower bus speed? If not, I still don't know why only one of my NICs ( each has two ethernet ports) is labeled as PCI-X in the output below. Does that mean it is running at PCI-X speeds are is it showing that it's capable?

# dmesg | grep e1000
[ 3678.349337] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI
[ 3678.349342] e1000: Copyright (c) 1999-2006 Intel Corporation.
[ 3678.349394] e1000 0000:06:07.0: PCI->APIC IRQ transform: INT A -> IRQ 48
[ 3678.409725] e1000 0000:06:07.0: Receive Descriptors set to 4096
[ 3678.409730] e1000 0000:06:07.0: Checksum Offload Disabled
[ 3678.409734] e1000 0000:06:07.0: Flow Control Disabled
[ 3678.586409] e1000 0000:06:07.0: eth0: (PCI:66MHz:32-bit) 00:11:43:e0:e2:8c
[ 3678.586419] e1000 0000:06:07.0: eth0: Intel(R) PRO/1000 Network Connection
[ 3678.586642] e1000 0000:07:08.0: PCI->APIC IRQ transform: INT A -> IRQ 49
[ 3678.649854] e1000 0000:07:08.0: Receive Descriptors set to 4096
[ 3678.649859] e1000 0000:07:08.0: Checksum Offload Disabled
[ 3678.649863] e1000 0000:07:08.0: Flow Control Disabled
[ 3678.826436] e1000 0000:07:08.0: eth1: (PCI:66MHz:32-bit) 00:11:43:e0:e2:8d
[ 3678.826444] e1000 0000:07:08.0: eth1: Intel(R) PRO/1000 Network Connection
[ 3678.826627] e1000 0000:09:04.0: PCI->APIC IRQ transform: INT A -> IRQ 82
[ 3679.093266] e1000 0000:09:04.0: Receive Descriptors set to 4096
[ 3679.093271] e1000 0000:09:04.0: Checksum Offload Disabled
[ 3679.093275] e1000 0000:09:04.0: Flow Control Disabled
[ 3679.130239] e1000 0000:09:04.0: eth2: (PCI-X:133MHz:64-bit) 00:04:23:e1:77:6a
[ 3679.130246] e1000 0000:09:04.0: eth2: Intel(R) PRO/1000 Network Connection
[ 3679.130449] e1000 0000:09:04.1: PCI->APIC IRQ transform: INT B -> IRQ 83
[ 3679.397312] e1000 0000:09:04.1: Receive Descriptors set to 4096
[ 3679.397318] e1000 0000:09:04.1: Checksum Offload Disabled
[ 3679.397321] e1000 0000:09:04.1: Flow Control Disabled
[ 3679.434350] e1000 0000:09:04.1: eth3: (PCI-X:133MHz:64-bit) 00:04:23:e1:77:6b
[ 3679.434360] e1000 0000:09:04.1: eth3: Intel(R) PRO/1000 Network Connection
[ 3679.434553] e1000 0000:0a:03.0: PCI->APIC IRQ transform: INT A -> IRQ 77
[ 3679.704072] e1000 0000:0a:03.0: Receive Descriptors set to 4096
[ 3679.704077] e1000 0000:0a:03.0: Checksum Offload Disabled
[ 3679.704081] e1000 0000:0a:03.0: Flow Control Disabled
[ 3679.738364] e1000 0000:0a:03.0: eth4: (PCI:33MHz:64-bit) 00:04:23:b6:35:6c
[ 3679.738371] e1000 0000:0a:03.0: eth4: Intel(R) PRO/1000 Network Connection
[ 3679.738538] e1000 0000:0a:03.1: PCI->APIC IRQ transform: INT B -> IRQ 78
[ 3680.046060] e1000 0000:0a:03.1: eth5: (PCI:33MHz:64-bit) 00:04:23:b6:35:6d
[ 3680.046067] e1000 0000:0a:03.1: eth5: Intel(R) PRO/1000 Network Connection
[ 3682.132415] e1000: eth0 NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
[ 3682.224423] e1000: eth1 NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
[ 3682.316385] e1000: eth2 NIC Link is Up 100 Mbps Half Duplex, Flow Control: None
[ 3682.408391] e1000: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 3682.500396] e1000: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None
[ 3682.708401] e1000: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX

At first I thought it was the NIC drivers but I'm not so sure. I really have no idea where else to look at the moment.

Any help is greatly appreciated as I'm struggling with this. If you need more information just ask.

Thanks!

[1]http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/Documentation/networking/e1000.txt?v=2.6.11.8 [2] http://support.dell.com/support/edocs/systems/pe2850/en/ug/t1390aa.htm