My challenge

I need to do tcpdumping of a lot of data - actually from 2 interfaces left in promiscuous mode that are able to see a lot of traffic.

To sum it up

• Log all traffic in promiscuous mode from 2 interfaces
• Those interfaces are not assigned an IP address
• pcap files must be rotated per ~1G
• When 10 TB of files are stored, start truncating the oldest

What I currently do

Right now I use tcpdump like this:

tcpdump -n -C 1000 -z /data/compress.sh -i any -w /data/livedump/capture.pcap $FILTER

The $FILTER contains src/dst filters so that I can use -i any. The reason for this is, that I have two interfaces and I would like to run the dump in a single thread rather than two.

compress.sh takes care of assigning tar to another CPU core, compress the data, give it a reasonable filename and move it to an archive location.

I cannot specify two interfaces, thus I have chosen to use filters and dump from any interface.

Right now, I do not do any housekeeping, but I plan on monitoring disk and when I have 100G left I will start wiping the oldest files - this should be fine.

And now; my problem

I see dropped packets. This is from a dump that has been running for a few hours and collected roughly 250 gigs of pcap files:

430083369 packets captured
430115470 packets received by filter
32057 packets dropped by kernel  <-- This is my concern

How can I avoid so many packets being dropped?

These things I did already try or look at

Changed the value of /proc/sys/net/core/rmem_max and /proc/sys/net/core/rmem_default which did indeed help - actually it took care of just around half of the dropped packets.

I have also looked at gulp - the problem with gulp is, that it does not support multiple interfaces in one process and it gets angry if the interface does not have an IP address. Unfortunately, that is a deal breaker in my case.

Next problem is, that when the traffic flows though a pipe, I cannot get the automatic rotation going. Getting one huge 10 TB file is not very efficient and I don't have a machine with 10TB+ RAM that I can run wireshark on, so that's out.

Do you have any suggestions? Maybe even a better way of doing my traffic dump altogether.